[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707816#comment-15707816 ] John Zhuge commented on HADOOP-13812: - Thx. > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707803#comment-15707803 ] Akira Ajisaka commented on HADOOP-13812: Now branch-2.6 is not maintained for Jenkins precommit job, so there are many -1's. > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13848) Missing auth-keys.xml prevents detecting test code build problem
[
https://issues.apache.org/jira/browse/HADOOP-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13848:
Priority: Blocker (was: Minor)
> Missing auth-keys.xml prevents detecting test code build problem
>
>
> Key: HADOOP-13848
> URL: https://issues.apache.org/jira/browse/HADOOP-13848
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3, fs/swift, test
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Blocker
>
> Both hadoop-aws and hadoop-openstack require the existence of file
> {{src/test/resources/auth-keys.xml}} to run the tests. With the design of the
> pom.xml, the non-existence of auth-keys.xml also prevents building the test
> code. Unfortunately this leads to delayed detection of build problems in test
> code, e.g., introduced by a mistake in backports.
> {code}
>
> tests-off
>
>
> src/test/resources/auth-keys.xml
>
>
>
> true
>
>
>
> tests-on
>
>
> src/test/resources/auth-keys.xml
>
>
>
> false
>
>
> {code}
> Section {{Skipping by Default}} in
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html
> proposes a solution. Any time you want to run tests, you must do 2 things
> instead of 1:
> * Copy auth-keys.xml to src/test/resources
> * Run {{mvn install}} with the extra {{-DskipTests=false}}
> Would like the community to weigh in on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13848) Missing auth-keys.xml prevents detecting test code build problem
[
https://issues.apache.org/jira/browse/HADOOP-13848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge updated HADOOP-13848:
Description:
Both hadoop-aws and hadoop-openstack require the existence of file
{{src/test/resources/auth-keys.xml}} to run the tests. With the design of the
pom.xml, the non-existence of auth-keys.xml also prevents building the test
code. Unfortunately this leads to delayed detection of build problems in test
code, e.g., introduced by a mistake in backports.
{code}
tests-off
src/test/resources/auth-keys.xml
true
tests-on
src/test/resources/auth-keys.xml
false
{code}
Section {{Skipping by Default}} in
http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html
proposes a solution. Any time you want to run tests, you must do 2 things
instead of 1:
* Copy auth-keys.xml to src/test/resources
* Run {{mvn install}} with the extra {{-DskipTests=false}}
Would like the community to weigh in on this.
was:
Both hadoop-aws and hadoop-openstack require the existence of file
{{src/test/resources/auth-keys.xml}} to run the tests. With the design of the
pom.xml, the non-existence of auth-keys.xml also prevents building the test
code. Unfortunately this leads to delayed detection of build problems in test
code, e.g., introduced by a mistake in backports.
{code}
tests-off
src/test/resources/auth-keys.xml
true
tests-on
src/test/resources/auth-keys.xml
false
{code}
Section {{Skipping by Default}} in
http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html
proposes a second solution. Any time you want to run tests, you must do 2
things instead of 1:
* Copy auth-keys.xml to src/test/resources
* Run {{mvn install}} with the extra {{-DskipTests=false}}
Would like the community to weigh in on this.
> Missing auth-keys.xml prevents detecting test code build problem
>
>
> Key: HADOOP-13848
> URL: https://issues.apache.org/jira/browse/HADOOP-13848
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3, fs/swift, test
>Reporter: John Zhuge
>Assignee: John Zhuge
>Priority: Minor
>
> Both hadoop-aws and hadoop-openstack require the existence of file
> {{src/test/resources/auth-keys.xml}} to run the tests. With the design of the
> pom.xml, the non-existence of auth-keys.xml also prevents building the test
> code. Unfortunately this leads to delayed detection of build problems in test
> code, e.g., introduced by a mistake in backports.
> {code}
>
> tests-off
>
>
> src/test/resources/auth-keys.xml
>
>
>
> true
>
>
>
> tests-on
>
>
> src/test/resources/auth-keys.xml
>
>
>
> false
>
>
> {code}
> Section {{Skipping by Default}} in
> http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html
> proposes a solution. Any time you want to run tests, you must do 2 things
> instead of 1:
> * Copy auth-keys.xml to src/test/resources
> * Run {{mvn install}} with the extra {{-DskipTests=false}}
> Would like the community to weigh in on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13675) Bug in return value for delete() calls in WASB
[
https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707722#comment-15707722
]
Hadoop QA commented on HADOOP-13675:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 11s{color} | {color:orange} hadoop-tools/hadoop-azure: The patch generated 4
new + 43 unchanged - 0 fixed = 47 total (was 43) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-azure in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13675 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840990/HADOOP-13675.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2fbda7bed30b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aeecfa2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11163/artifact/patchprocess/diff-checkstyle-hadoop-tools_hadoop-azure.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11163/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11163/artifact/patchprocess/patch-asflicense-problems.txt
|
| modules | C: hadoop-tools/hadoop-azure U: hadoop-tools/hadoop-azure |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11163/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Bug in return value for delete() calls in WASB
> --
>
> Key: HADOOP-13675
> URL: https://issues.apache.org/jira/browse/HADOOP-13675
> Project: Hadoop Common
> Issue Type: Bug
>
[jira] [Comment Edited] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth edited comment on HADOOP-13675 at 11/30/16 6:17 AM: -- [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on, kindly review the patch. was (Author: dchickabasapa): [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth edited comment on HADOOP-13675 at 11/30/16 6:17 AM: -- [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I have attached Hadoop-13675.002.patch that Jameel had worked on. was (Author: dchickabasapa): [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I am attaching Patch-2 that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth updated HADOOP-13675: --- Attachment: HADOOP-13675.002.patch > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch, HADOOP-13675.002.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707688#comment-15707688 ] Dushyanth commented on HADOOP-13675: [~liuml07] Thanks. I will be taking over the responsibility of uploading the second patch and addressing any issues for this Jira going forward. I am attaching Patch-2 that Jameel had worked on. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10930) HarFsInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HADOOP-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707686#comment-15707686
]
Hadoop QA commented on HADOOP-10930:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 1 new + 42 unchanged - 0 fixed = 43 total (was 42) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-10930 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12659535/HADOOP-10930.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e4002e8c7066 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / aeecfa2 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11162/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11162/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11162/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11162/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> HarFsInputStream should implement
[jira] [Assigned] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dushyanth reassigned HADOOP-13675: -- Assignee: Dushyanth (was: Jameel Naina Mohamed) > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13848) Missing auth-keys.xml prevents detecting test code build problem
John Zhuge created HADOOP-13848:
---
Summary: Missing auth-keys.xml prevents detecting test code build
problem
Key: HADOOP-13848
URL: https://issues.apache.org/jira/browse/HADOOP-13848
Project: Hadoop Common
Issue Type: Improvement
Components: fs/s3, fs/swift, test
Reporter: John Zhuge
Assignee: John Zhuge
Priority: Minor
Both hadoop-aws and hadoop-openstack require the existence of file
{{src/test/resources/auth-keys.xml}} to run the tests. With the design of the
pom.xml, the non-existence of auth-keys.xml also prevents building the test
code. Unfortunately this leads to delayed detection of build problems in test
code, e.g., introduced by a mistake in backports.
{code}
tests-off
src/test/resources/auth-keys.xml
true
tests-on
src/test/resources/auth-keys.xml
false
{code}
Section {{Skipping by Default}} in
http://maven.apache.org/surefire/maven-surefire-plugin/examples/skipping-test.html
proposes a second solution. Any time you want to run tests, you must do 2
things instead of 1:
* Copy auth-keys.xml to src/test/resources
* Run {{mvn install}} with the extra {{-DskipTests=false}}
Would like the community to weigh in on this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707670#comment-15707670 ] Sahil Takiar commented on HADOOP-13600: --- Sounds good Steve. I rebased the PR and created a separate TransferManager just for copies. Haven't looked at the testing yet. > S3a rename() to copy files in a directory in parallel > - > > Key: HADOOP-13600 > URL: https://issues.apache.org/jira/browse/HADOOP-13600 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Sahil Takiar > > Currently a directory rename does a one-by-one copy, making the request > O(files * data). If the copy operations were launched in parallel, the > duration of the copy may be reducable to the duration of the longest copy. > For a directory with many files, this will be significant -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10930) HarFsInputStream should implement PositionedReadable with thead-safe.
[
https://issues.apache.org/jira/browse/HADOOP-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707573#comment-15707573
]
Hudson commented on HADOOP-10930:
-
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #10912 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/10912/])
HADOOP-10930. Refactor: Wrap Datanode IO related operations. Contributed (arp:
rev aeecfa24f4fb6af289920cbf8830c394e66bd78e)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaOutputStreams.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplicaInPipeline.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestSimulatedFSDataset.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalDatasetImpl.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileAppend.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/LocalReplica.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/ReplicaInputStreams.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaInPipeline.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetAsyncDiskService.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockSender.java
> HarFsInputStream should implement PositionedReadable with thead-safe.
> -
>
> Key: HADOOP-10930
> URL: https://issues.apache.org/jira/browse/HADOOP-10930
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.6.0
>Reporter: Yi Liu
>Assignee: Yi Liu
> Labels: BB2015-05-TBR
> Attachments: HADOOP-10930.001.patch
>
>
> {{PositionedReadable}} definition requires the implementations for its
> interfaces should be thread-safe.
> HarFsInputStream doesn't implement these interfaces with tread-safe, this
> JIRA is to fix this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13837) Process check bug in hadoop_stop_daemon of hadoop-functions.sh
[
https://issues.apache.org/jira/browse/HADOOP-13837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated HADOOP-13837:
-
Target Version/s: 3.0.0-alpha2
> Process check bug in hadoop_stop_daemon of hadoop-functions.sh
> --
>
> Key: HADOOP-13837
> URL: https://issues.apache.org/jira/browse/HADOOP-13837
> Project: Hadoop Common
> Issue Type: Bug
> Components: scripts
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: HADOOP-13837.01.patch, HADOOP-13837.02.patch,
> HADOOP-13837.03.patch, HADOOP-13837.04.patch, check_proc.sh
>
>
> Always get {{ERROR: Unable to kill ...}} after {{Trying to kill with kill
> -9}}, see following output of stop-yarn.sh
> {code}
> : WARNING: nodemanager did not stop gracefully after 5 seconds:
> Trying to kill with kill -9
> : ERROR: Unable to kill 18097
> {code}
> hadoop_stop_daemon doesn't check process liveness correctly, this bug can be
> reproduced by the script easily. kill -9 would need some time to be done,
> directly check process existence right after mostly will fail.
> {code}
> function hadoop_stop_daemon
> {
> ...
> kill -9 "${pid}" >/dev/null 2>&1
> fi
> if ps -p "${pid}" > /dev/null 2>&1; then
> hadoop_error "ERROR: Unable to kill ${pid}"
> else
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704286#comment-15704286
]
John Zhuge edited comment on HADOOP-13597 at 11/30/16 2:58 AM:
---
Map {{sbin/kms.sh}} to {{bin/hadoop kms}}:
| kms.sh run | hadoop kms |
| kms.sh start | hadoop --daemon start kms |
| kms.sh stop | hadoop --daemon stop kms |
| kms.sh status | hadoop --daemon status kms |
was (Author: jzhuge):
Map {{sbin/kms.sh}} to {{bin/hadoop kms}}:
| kms.sh run | hadoop kms |
| kms.sh start | hadoop kms --daemon start |
| kms.sh stop | hadoop kms --daemon stop |
| kms.sh status | hadoop kms --daemon status |
> Switch KMS from Tomcat to Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-13597.001.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-13847) KMSWebApp should close KeyProviderCryptoExtension
[ https://issues.apache.org/jira/browse/HADOOP-13847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge reassigned HADOOP-13847: --- Assignee: John Zhuge > KMSWebApp should close KeyProviderCryptoExtension > - > > Key: HADOOP-13847 > URL: https://issues.apache.org/jira/browse/HADOOP-13847 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Reporter: Anthony Young-Garner >Assignee: John Zhuge > > KeyProviderCryptoExtension should be closed so that all KeyProviders are also > closed. See related HADOOP-13838. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707310#comment-15707310
]
Aaron Fabbri commented on HADOOP-13449:
---
This fixes that failed unit test for me:
{code}
---
a/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java
+++
b/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/MockS3ClientFactory.java
@@ -41,9 +41,8 @@
@Override
public AmazonDynamoDBClient createDynamoDBClient(
URI uri, com.amazonaws.regions.Region region) throws IOException {
-final DefaultS3ClientFactory factory = new DefaultS3ClientFactory();
-factory.setConf(getConf());
-return factory.createDynamoDBClient(uri, region);
+throw new IOException("Purposely failing to create DynamoDB client"
+ + " for unit test.");
}
{code}
Also noticed a spot we need to fix the exception thrown (supposed to be an
IOException):
{code}
@Override
public AmazonDynamoDBClient createDynamoDBClient(URI fsUri, Region region)
throws IOException {
{code}
...
{code}
String msg = "Incorrect DynamoDB endpoint: " + endPoint;
LOG.error(msg, e);
throw new IllegalArgumentException(msg, e);
}
}
{code}
I have a number of integration test failures I'll be working through next.
BTW, I'm happy to submit a follow-up (v11) patch with these things if that
would help, just shout.
> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch,
> HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch,
> HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707232#comment-15707232
]
Aaron Fabbri commented on HADOOP-13449:
---
Congrats on the clean jenkins run [~liuml07].
I'm running some tests with this configured as the MetadataStore today. First
thing I'm noticing is a failure in {{TestS3AGetFileStatus}}. This led me to
this part of the v10 patch:
{code}
+public class MockS3ClientFactory extends Configured implements S3ClientFactory
{
+
+ @Override
+ public AmazonDynamoDBClient createDynamoDBClient(
+ URI uri, com.amazonaws.regions.Region region) throws IOException {
+final DefaultS3ClientFactory factory = new DefaultS3ClientFactory();
+factory.setConf(getConf());
+return factory.createDynamoDBClient(uri, region);
+ }
{code}
I believe the goal of the mock s3 client is to be able to run non-integration
(unit) tests without S3 configured. It looks like you are creating an actual
S3 client from the mock client. Doesn't this break the ability of unit tests
to run without S3?
It seems like all the unit tests should only use MetadataStores which can run
locally (Null or LocalMetadataStore). So, maybe we do not need this code at
all. Maybe MockS3ClientFactory#createDynamoDBClient() just throws a runtime
exception "Failing to create DynamoDB for unit test", and then we fall back to
the NullMetadataStore automatically in S3Guard#getMetadataStore()?
I'm also wondering if, instead of having S3ClientFactory expose a
createDynamoDBClient() method, we should just add getters to S3ClientFactory
(getAwsConfig() and maybe getCredentials()), and then move
createDynamoDBClient() to inside the DynamoDBMetadataStore? The
DynamoDBMetadataStore can then call the getters on the client to get what it
needs to construct a DynamoDB client. The goal here would be to keep dynamodb
specifics encapsulated in DynamoDBMetadataStore. This would allow, for
example, removing the Dynamo dependency from s3a if we ever want to create a
separate submodule for the DynamoDBMetadataStore.
> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch,
> HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch,
> HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13818) TestLocalFileSystem#testSetTimes fails
[
https://issues.apache.org/jira/browse/HADOOP-13818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707194#comment-15707194
]
Yiqun Lin commented on HADOOP-13818:
The Jenkins result seems clean. I agree with the opinion that we should
minimise use of the call {{getFileStatus}} since these calls as potentially
very expensive in some FS. What do you think, [~ajisakaa]?
> TestLocalFileSystem#testSetTimes fails
> --
>
> Key: HADOOP-13818
> URL: https://issues.apache.org/jira/browse/HADOOP-13818
> Project: Hadoop Common
> Issue Type: Bug
> Components: test
> Environment: Mac OS Sierra, OpenJDK 8u122-ea
>Reporter: Akira Ajisaka
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HADOOP-13818.001.patch, HADOOP-13818.002.patch,
> HADOOP-13818.003.patch
>
>
> {noformat}
> Running org.apache.hadoop.fs.TestLocalFileSystem
> Tests run: 20, Failures: 1, Errors: 0, Skipped: 1, Time elapsed: 4.887 sec
> <<< FAILURE! - in org.apache.hadoop.fs.TestLocalFileSystem
> testSetTimes(org.apache.hadoop.fs.TestLocalFileSystem) Time elapsed: 0.084
> sec <<< FAILURE!
> java.lang.AssertionError: expected:<23456000> but was:<1479176144000>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.fs.TestLocalFileSystem.checkTimesStatus(TestLocalFileSystem.java:391)
> at
> org.apache.hadoop.fs.TestLocalFileSystem.testSetTimes(TestLocalFileSystem.java:414)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12329) io.file.buffer.size is only for SequenceFiles
[
https://issues.apache.org/jira/browse/HADOOP-12329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707138#comment-15707138
]
Andres Perez commented on HADOOP-12329:
---
I think this can be easily be solved by removing the first setence and
reorganizing a little bit the description to look like this:
{code:xml}
Determines how much data is buffered during read and write
operations. The size of this buffer should probably be a multiple of
hardware page size (4096 on Intel x86).
{code}
> io.file.buffer.size is only for SequenceFiles
> -
>
> Key: HADOOP-12329
> URL: https://issues.apache.org/jira/browse/HADOOP-12329
> Project: Hadoop Common
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.7.1
>Reporter: Kun Yan
>Priority: Trivial
> Labels: documentation
>
> the core-site.xml io.file.buffer.size description:Size of read/write buffer
> used in SequenceFiles.
> This parameter is only for SequenceFiles? I search for the other one issue as
> that is not true? If not only affect SequenceFiles could let users to ignore
> this parameter.URL
> location:http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/ClusterSetup.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707117#comment-15707117
]
Xiao Chen commented on HADOOP-13597:
Thanks for working on this [~jzhuge].
Some review comments aside from your todos:
- Suggest to name the new config key names _KEY, to be more consistent with
HDFS/Common way.
- Nice we're loading envvars for compatibility. It'll make upgrades less
painful. OTOH, could you add some text in the {{Loaded env}} logs to indicate
this is a deprecated way, and user should user configuration files instead?
- {{kms-default.xml}} has some duplicate key names
(hadoop.kms.authentication.signer.secret.provider.zookeeper.XXX).
- This exists before your patch, but good to fix along: kms-default.xml has
some escaped chars () in configuration.
- {{hadoop.kms.authentication.signer.secret.provider.zookeeper.auth.type}}
defaults to {{none}} before. We should keep that unchanged. The {{kerberos}}
default value also doesn't work, it should be {{sasl}}
- We should add basic testing to {{KMSHttpServer}}
- Had a quick run, kms.log seems record every request/response. By default this
shouldn't be logged.
- After clean install kms.log appears to be at TRACE level. Suggest to make it
INFO.
> Switch KMS from Tomcat to Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-13597.001.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13449) S3Guard: Implement DynamoDBMetadataStore.
[
https://issues.apache.org/jira/browse/HADOOP-13449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707088#comment-15707088
]
Mingliang Liu commented on HADOOP-13449:
I merged again from the {{trunk}} locally and resolved minor conflicts with
[HADOOP-13823]. If you guys support the merge, I'll push it to the remote
feature branch.
> S3Guard: Implement DynamoDBMetadataStore.
> -
>
> Key: HADOOP-13449
> URL: https://issues.apache.org/jira/browse/HADOOP-13449
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Mingliang Liu
> Attachments: HADOOP-13449-HADOOP-13345.000.patch,
> HADOOP-13449-HADOOP-13345.001.patch, HADOOP-13449-HADOOP-13345.002.patch,
> HADOOP-13449-HADOOP-13345.003.patch, HADOOP-13449-HADOOP-13345.004.patch,
> HADOOP-13449-HADOOP-13345.005.patch, HADOOP-13449-HADOOP-13345.006.patch,
> HADOOP-13449-HADOOP-13345.007.patch, HADOOP-13449-HADOOP-13345.008.patch,
> HADOOP-13449-HADOOP-13345.009.patch, HADOOP-13449-HADOOP-13345.010.patch
>
>
> Provide an implementation of the metadata store backed by DynamoDB.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13675: --- Assignee: (was: Dushyanth) > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706925#comment-15706925 ] Mingliang Liu commented on HADOOP-13675: Hi [~jameeln], I have added you to the Hadoop Contributor on this site; and assigned this JIRA to you. [~dchickabasapa] please feel free to re-assign to yourself after Jameel uploads the new patch. > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Jameel Naina Mohamed > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13675) Bug in return value for delete() calls in WASB
[ https://issues.apache.org/jira/browse/HADOOP-13675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingliang Liu updated HADOOP-13675: --- Assignee: Jameel Naina Mohamed > Bug in return value for delete() calls in WASB > -- > > Key: HADOOP-13675 > URL: https://issues.apache.org/jira/browse/HADOOP-13675 > Project: Hadoop Common > Issue Type: Bug > Components: azure, fs/azure >Affects Versions: 2.8.0 >Reporter: Dushyanth >Assignee: Jameel Naina Mohamed > Fix For: 2.9.0 > > Attachments: HADOOP-13675.001.patch > > > Current implementation of WASB does not correctly handle multiple > threads/clients calling delete on the same file. The expected behavior in > such scenarios is only one of the thread should delete the file and return > true, while all other threads should receive false. However in the current > implementation even though only one thread deletes the file, multiple clients > incorrectly get "true" as the return from delete() call.. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13824) FsShell can suppress the real error if no error message is present
[
https://issues.apache.org/jira/browse/HADOOP-13824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706668#comment-15706668
]
Wei-Chiu Chuang commented on HADOOP-13824:
--
[~jzhuge] thanks for the patch. I reviewed it and have just spotted two very
tiny nits. Both are subjective to my personal preference:
* {{StringContains#containsString}} may be replaced by
{{{GenericTestUtils#assertMatches}}.
* I feel {{SystemErrCapture}} should go into {{GenericTestUtils}}. There's a
{{LogCapturer}} class in {{GenericTestUtils}} serving the similar purpose for
logs. But that's purely my personal preference.
> FsShell can suppress the real error if no error message is present
> --
>
> Key: HADOOP-13824
> URL: https://issues.apache.org/jira/browse/HADOOP-13824
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1, 2.7.3
>Reporter: Rob Vesse
>Assignee: John Zhuge
> Labels: supportability
> Attachments: HADOOP-13824.001.patch, HADOOP-13824.002.patch
>
>
> The {{FsShell}} error handling assumes in {{displayError()}} that the
> {{message}} argument is not {{null}}. However in the case where it is this
> leads to a NPE which results in suppressing the actual error information
> since a higher level of error handling kicks in and just dumps the stack
> trace of the NPE instead.
> e.g.
> {noformat}
> Exception in thread "main" java.lang.NullPointerException
> at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
> at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
> {noformat}
> This is deeply unhelpful because depending on what the underlying error was
> there may be no stack dumped/logged for it (as HADOOP-7114 provides) since
> {{FsShell}} doesn't explicitly dump traces for {{IllegalArgumentException}}
> which appears to be the underlying cause of my issue. Line 289 is where
> {{displayError()}} is called for {{IllegalArgumentException}} handling and
> that catch clause does not log the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
[
https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706483#comment-15706483
]
Mingliang Liu commented on HADOOP-13846:
This proposal makes a lot of sense. My only concern is the FS#rename/3 is
marked deprecated, and users may think we don't support this in any future.
> S3A to implement rename(final Path src, final Path dst, final Rename...
> options)
>
>
> Key: HADOOP-13846
> URL: https://issues.apache.org/jira/browse/HADOOP-13846
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> S3a now raises exceptions on invalid rename operations, but these get lost. I
> plan to use them in my s3guard committer HADOOP-13786.
> Rather than just make innerRename() private, S3A could implement
> {{FileSystem.rename(final Path src, final Path dst, final Rename...
> options)}} and so have an exception-raising rename which can be called
> without going more into the internals.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13847) KMSWebApp should close KeyProviderCryptoExtension
Anthony Young-Garner created HADOOP-13847: - Summary: KMSWebApp should close KeyProviderCryptoExtension Key: HADOOP-13847 URL: https://issues.apache.org/jira/browse/HADOOP-13847 Project: Hadoop Common Issue Type: Bug Components: kms Reporter: Anthony Young-Garner KeyProviderCryptoExtension should be closed so that all KeyProviders are also closed. See related HADOOP-13838. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13840) Implement getUsed() for ViewFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706437#comment-15706437
]
Hadoop QA commented on HADOOP-13840:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
49s{color} | {color:red} hadoop-common-project_hadoop-common generated 3 new +
0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
34s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 48m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840911/HADOOP-13840.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3b1ea88171bc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b407d53 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| javadoc |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11161/artifact/patchprocess/diff-javadoc-javadoc-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11161/testReport/ |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11161/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Implement getUsed() for ViewFileSystem
> --
>
> Key: HADOOP-13840
> URL: https://issues.apache.org/jira/browse/HADOOP-13840
> Project: Hadoop Common
> Issue Type: Task
> Components: viewfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee:
[jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[
https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706370#comment-15706370
]
Steve Loughran commented on HADOOP-13600:
-
sahil, unless you object, I'm assigning this to you; I'll help with review &
test. I do like the idea of fault injection via a new s3 client though.
BTW, filed HADOOP-13846; implementing the {{FileSystem.rename(final Path src,
final Path dst, final Rename... options)}} method. That can come after; I'm
just thinking it may be the way to expose exception raising in rename()
operations.
> S3a rename() to copy files in a directory in parallel
> -
>
> Key: HADOOP-13600
> URL: https://issues.apache.org/jira/browse/HADOOP-13600
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Steve Loughran
>Assignee: Sahil Takiar
>
> Currently a directory rename does a one-by-one copy, making the request
> O(files * data). If the copy operations were launched in parallel, the
> duration of the copy may be reducable to the duration of the longest copy.
> For a directory with many files, this will be significant
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
[
https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706366#comment-15706366
]
Steve Loughran commented on HADOOP-13846:
-
Requirements
# rename/3 behaviour to be specified fully. It's better than rename/2 here.
# contract tests
# S3a to implement with contract tests
# have rename/2 forward to the new rename, catching IOEs and downgrading to
return codes, the way it now does. Are behaviours consistent enough to do that?
> S3A to implement rename(final Path src, final Path dst, final Rename...
> options)
>
>
> Key: HADOOP-13846
> URL: https://issues.apache.org/jira/browse/HADOOP-13846
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>
> S3a now raises exceptions on invalid rename operations, but these get lost. I
> plan to use them in my s3guard committer HADOOP-13786.
> Rather than just make innerRename() private, S3A could implement
> {{FileSystem.rename(final Path src, final Path dst, final Rename...
> options)}} and so have an exception-raising rename which can be called
> without going more into the internals.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)
Steve Loughran created HADOOP-13846:
---
Summary: S3A to implement rename(final Path src, final Path dst,
final Rename... options)
Key: HADOOP-13846
URL: https://issues.apache.org/jira/browse/HADOOP-13846
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
S3a now raises exceptions on invalid rename operations, but these get lost. I
plan to use them in my s3guard committer HADOOP-13786.
Rather than just make innerRename() private, S3A could implement
{{FileSystem.rename(final Path src, final Path dst, final Rename... options)}}
and so have an exception-raising rename which can be called without going more
into the internals.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13786) add output committer which uses s3guard for consistent O(1) commits to S3
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706351#comment-15706351
]
Steve Loughran edited comment on HADOOP-13786 at 11/29/16 7:51 PM:
---
For the curious, my WiP, in the s3guard branch, is here:
https://github.com/steveloughran/hadoop/tree/s3guard/HADOOP-13786-committer
I've been tweaking the mapreduce code to add ability to switch to a subclass of
FileOutputCommitter in FileOutputFormat (And transitively, all children), by
way of a factory. The S3a factory dynamically chooses the committer based on
the destination FS. There's currently no difference in the codebase apart from
logging of operation durations. This means that we can switch the committer
behind all output formats to using the s3a one (and also allowing anyone else
to do the same with their own FOF committer subclass).
I can see that ParquetOutputFormat has its own committer,
{{ParquetOutputCommitter}}, as used by {{ParquetOutputFormat}} and so, I
believe spark's {{dataframe.parquet.write()}} pipeline; my IDE isn't
highlighting much else.
Now, could I get away with just modifying the base FOF committer?
for exception reporting it could use {{FileSystem.rename(final Path src, final
Path dst, final Rename... options)}}; S3A to impl that with exceptions. That's
the "transitory" method for use between FS and FileContext, which does raise
exceptions; the default is non-atomic and eventually gets to {{rename(src,
dest)}}, except for HDFS which does a full implementation. But: what if the
semantics of that rename() (not yet in the FS spec, AFAIK) are different from
what callers expect in some of the corner cases? And so commits everywhere
break? That would not be good. And it would remove the ability to use any
private s3a/s3guard calls if we wanted some (e.g. get a lock on a directory),
unless they went in to the standard FS APIs.
Making the output factory pluggable would avoid such problems, though logging
duration might be nice all round. That said, given the time to rename, its
somewhat unimportant when using real filesystems.
was (Author: ste...@apache.org):
For the curious, my WiP, in the s3guard branch, is here:
https://github.com/steveloughran/hadoop/tree/s3guard/HADOOP-13786-committer
I've been tweaking the mapreduce code to add ability to switch to a subclass of
FileOutputCommitter in FileOutputFormat (And transitively, all children), by
way of a factory. The S3a factory dynamically chooses the committer based on
the destination FS. There's currently no difference in the codebase apart from
logging of operation durations. This means that we can switch the committer
behind all output formats to using the s3a one (and also allowing anyone else
to do the same with their own FOF committer subclass).
I can see that ParquetOutputFormat has its own committer,
{{ParquetOutputCommitter}}, as used by {{ParquetOutputFormat}} and so, I
believe spark's {{dataframe.parquet.write()}} pipeline; my IDE isn't
highlighting much else.
Now, could I get away with just modifying the base FOF committer?
for exception reporting it could use {{FileSystem.rename(final Path src, final
Path dst, final Rename... options)}}; S3A to impl that with exceptions. That's
the "transitory" method for use between FS and FileContext, which does raise
exceptions; the default is non-atomic and eventually gets to {{rename(src,
dest)}}, except for HDFS which does a full implementation. But: what if the
semantics of that rename() (not yet in the FS spec, AFAIK) are different from
what callers expect in some of the corner cases? And so commits everywhere
break? That would not be good. And it would remove the ability to use any
private s3a/s3guard calls if we wanted some (e.g. get a lock on a directory),
unless they went in to the standard FS APIs.
Making the output factory pluggable would avoid such problems, though logging
duration might be nice all round. That said, given the time to rename, its
somewhat unimportant when using real filesystems.
> add output committer which uses s3guard for consistent O(1) commits to S3
> -
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to
[jira] [Commented] (HADOOP-13786) add output committer which uses s3guard for consistent O(1) commits to S3
[
https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706351#comment-15706351
]
Steve Loughran commented on HADOOP-13786:
-
For the curious, my WiP, in the s3guard branch, is here:
https://github.com/steveloughran/hadoop/tree/s3guard/HADOOP-13786-committer
I've been tweaking the mapreduce code to add ability to switch to a subclass of
FileOutputCommitter in FileOutputFormat (And transitively, all children), by
way of a factory. The S3a factory dynamically chooses the committer based on
the destination FS. There's currently no difference in the codebase apart from
logging of operation durations. This means that we can switch the committer
behind all output formats to using the s3a one (and also allowing anyone else
to do the same with their own FOF committer subclass).
I can see that ParquetOutputFormat has its own committer,
{{ParquetOutputCommitter}}, as used by {{ParquetOutputFormat}} and so, I
believe spark's {{dataframe.parquet.write()}} pipeline; my IDE isn't
highlighting much else.
Now, could I get away with just modifying the base FOF committer?
for exception reporting it could use {{FileSystem.rename(final Path src, final
Path dst, final Rename... options)}}; S3A to impl that with exceptions. That's
the "transitory" method for use between FS and FileContext, which does raise
exceptions; the default is non-atomic and eventually gets to {{rename(src,
dest)}}, except for HDFS which does a full implementation. But: what if the
semantics of that rename() (not yet in the FS spec, AFAIK) are different from
what callers expect in some of the corner cases? And so commits everywhere
break? That would not be good. And it would remove the ability to use any
private s3a/s3guard calls if we wanted some (e.g. get a lock on a directory),
unless they went in to the standard FS APIs.
Making the output factory pluggable would avoid such problems, though logging
duration might be nice all round. That said, given the time to rename, its
somewhat unimportant when using real filesystems.
> add output committer which uses s3guard for consistent O(1) commits to S3
> -
>
> Key: HADOOP-13786
> URL: https://issues.apache.org/jira/browse/HADOOP-13786
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.0.0-alpha2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> A goal of this code is "support O(1) commits to S3 repositories in the
> presence of failures". Implement it, including whatever is needed to
> demonstrate the correctness of the algorithm. (that is, assuming that s3guard
> provides a consistent view of the presence/absence of blobs, show that we can
> commit directly).
> I consider ourselves free to expose the blobstore-ness of the s3 output
> streams (ie. not visible until the close()), if we need to use that to allow
> us to abort commit operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13794) JSON.org license is now CatX
[
https://issues.apache.org/jira/browse/HADOOP-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HADOOP-13794:
-
Affects Version/s: (was: 3.0.0-alpha2)
I'm going to remove the 3.0.0-alpha2 affects version since the AWS and jackson
deps have been upgraded in trunk
This is also just a stay of execution. We can keep spinning out maintenance
releases until April, but we really need to get 2.8 (and 2.9?) out so we can
retire the 2.6 and 2.7 lines.
> JSON.org license is now CatX
>
>
> Key: HADOOP-13794
> URL: https://issues.apache.org/jira/browse/HADOOP-13794
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.4, 2.6.6
>Reporter: Sean Busbey
>Priority: Blocker
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list.
> Prior to this, use of the JSON Java library was allowed. See Debian's page
> for a list of alternatives.
> {quote}
> We have a test-time transitive dependency on the {{org.json:json}} artifact
> in trunk and branch-2. AFAICT, this test time dependency doesn't get exposed
> to downstream at all (I checked assemblies and test-jar artifacts we publish
> to maven), so it can be removed or kept at our leisure. keeping it risks it
> being promoted out of test scope by maven without us noticing. We might be
> able to add an enforcer rule to check for this.
> We also distribute it in bundled form through our use of the AWS Java SDK
> artifacts in trunk and branch-2. Looking at the github project, [their
> dependency on JSON.org was removed in
> 1.11|https://github.com/aws/aws-sdk-java/pull/417], so if we upgrade to
> 1.11.0+ we should be good to go. (this might be hard in branch-2.6 and
> branch-2.7 where we're on 1.7.4)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13840) Implement getUsed() for ViewFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manoj Govindassamy updated HADOOP-13840:
Attachment: HADOOP-13840.02.patch
Thanks [~ste...@apache.org]. Yes, assertEquals can give us verbose information
on the comparisons done. Updated the patch with Assert.assertTrue replaced with
assertEquals.
> Implement getUsed() for ViewFileSystem
> --
>
> Key: HADOOP-13840
> URL: https://issues.apache.org/jira/browse/HADOOP-13840
> Project: Hadoop Common
> Issue Type: Task
> Components: viewfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-13840.01.patch, HADOOP-13840.02.patch
>
>
> ViewFileSystem doesn't override {{FileSystem#getUSed()}}. So, when file
> system used space is queried for slash root "/" paths, the default
> implementations tries to run the {{getContentSummary}} which crashes on
> seeing unexpected mount points under slash.
> ViewFileSystem#getUsed() is not expected to collate all the space used from
> all the mount points configured under "/". Proposal is to avoid invoking
> FileSystem#getUsed() and throw NotInMountPointException until LinkMergeSlash
> is supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706266#comment-15706266
]
Luke Miner edited comment on HADOOP-13811 at 11/29/16 7:23 PM:
---
Turns out I still had {{hadoop-aws:2.7.3}} in my spark conf file. I ended up
including the {{hadoop-aws-2.9.0-SNAPSHOT.jar}} that I built using the
instructions you gave above. I also bumped up amazon's {{aws-java-sdk}} to
{{1.11.57}}. I'm still seeing the same error, only on a different line number
now. Oddly, it also seems to be telling me that I should be using
{{S3AFileSystem}} instead of {{S3FileSystem}}.
{code}
S3FileSystem is deprecated and will be removed in future releases. Use
NativeS3FileSystem or S3AFileSystem instead.
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 177.7 KB, free 366.1 MB)
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 21.0 KB, free 366.1 MB)
16/11/29 19:00:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.0.229.45:52703 (size: 21.0 KB, free: 366.3 MB)
16/11/29 19:00:19 INFO SparkContext: Created broadcast 0 from textFile at
json2pq.scala:130
Exception in thread "main" java.lang.NumberFormatException: For input string:
"100M"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1320)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2904)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:101)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2941)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2923)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:265)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1957)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:928)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
at org.apache.spark.rdd.RDD.collect(RDD.scala:927)
at Json2Pq$.main(json2pq.scala:130)
at Json2Pq.main(json2pq.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
I did also try to run it using {{hadoop-aws-3.0.0-alpha1.jar}} that is
currently on central, but got this error instead. Perhaps because I'm running
off a hadoop 2.9 snapshot.
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Error while
instantiating 'org.apache.spark.sql.internal.SessionState':
at
org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:965)
at
[jira] [Comment Edited] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706266#comment-15706266
]
Luke Miner edited comment on HADOOP-13811 at 11/29/16 7:22 PM:
---
Turns out I still had {{hadoop-aws:2.7.3}} in my spark conf file. I ended up
including the {{hadoop-aws-2.9.0-SNAPSHOT.jar}} that I built using the
instructions you gave above. I also bumped up amazon's {{aws-java-sdk}} to
{{1.11.57}}. I'm still seeing the same error, only on a different line number
now. Oddly, it also seems to be telling me that I should be using
{{S3AFileSystem}} instead of {{S3FileSystem}}.
{code}
S3FileSystem is deprecated and will be removed in future releases. Use
NativeS3FileSystem or S3AFileSystem instead.
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 177.7 KB, free 366.1 MB)
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 21.0 KB, free 366.1 MB)
16/11/29 19:00:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.0.229.45:52703 (size: 21.0 KB, free: 366.3 MB)
16/11/29 19:00:19 INFO SparkContext: Created broadcast 0 from textFile at
json2pq.scala:130
Exception in thread "main" java.lang.NumberFormatException: For input string:
"100M"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1320)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2904)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:101)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2941)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2923)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:265)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1957)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:928)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
at org.apache.spark.rdd.RDD.collect(RDD.scala:927)
at Json2Pq$.main(json2pq.scala:130)
at Json2Pq.main(json2pq.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
I did also try to run it using {{hadoop-aws-3.0.0-alpha1.jar}} that is
currently on central, but got this error instead. Perhaps because I'm running
off a hadoop 2.9 snapshot.
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Error while
instantiating 'org.apache.spark.sql.internal.SessionState':
at
org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:965)
at
[jira] [Commented] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706266#comment-15706266
]
Luke Miner commented on HADOOP-13811:
-
Turns out I still had {{hadoop-aws:2.7.3}} in my spark conf file. I ended up
including the {{hadoop-aws-2.9.0-SNAPSHOT.jar}} that I built using the
instructions you gave above. I also bumped up amazon's {{aws-java-sdk}} to
{{1.11.57}}. I'm still seeing the same error, only on a different line number
now. Oddly, it also seems to be telling me that I should be using
{{S3AFileSystem}} instead of {{S3FileSystem}}.
{code}
S3FileSystem is deprecated and will be removed in future releases. Use
NativeS3FileSystem or S3AFileSystem instead.
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0 stored as values in
memory (estimated size 177.7 KB, free 366.1 MB)
16/11/29 19:00:19 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in
memory (estimated size 21.0 KB, free 366.1 MB)
16/11/29 19:00:19 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on
10.0.229.45:52703 (size: 21.0 KB, free: 366.3 MB)
16/11/29 19:00:19 INFO SparkContext: Created broadcast 0 from textFile at
json2pq.scala:130
Exception in thread "main" java.lang.NumberFormatException: For input string:
"100M"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Long.parseLong(Long.java:441)
at java.lang.Long.parseLong(Long.java:483)
at org.apache.hadoop.conf.Configuration.getLong(Configuration.java:1320)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:234)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2904)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:101)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2941)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2923)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:390)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
at
org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:265)
at
org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236)
at
org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at
org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:248)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:246)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:246)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1957)
at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:928)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
at org.apache.spark.rdd.RDD.collect(RDD.scala:927)
at Json2Pq$.main(json2pq.scala:130)
at Json2Pq.main(json2pq.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}
I did also try to run it using {{hadoop-aws-3.0.0-alpha1.jar}} that is
currently on central, but got this error instead. Perhaps because I'm running
off a hadoop 2.9 snapshot.
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Error while
instantiating 'org.apache.spark.sql.internal.SessionState':
at
org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:965)
at
org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
at
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706113#comment-15706113
]
Allen Wittenauer commented on HADOOP-13650:
---
Based upon this patch, you don't need separate scripts/separate command.
There's not enough here to warrant polluting the shell path. (The single case
statement is a dead give away.) Use a shell profile to add the s3guard command
to bin/hadoop dynamically. See, e.g., distcp, hadoop-archives, etc.
Also:
{code}
if declare -f hadoop_subcommand_
{code}
(plus a few other spots)
This isn't bin/hadoop, so doing this will conflict.
{code}
+hadoop_debug "Appending hadoop-aws jars"
+hadoop_add_classpath "${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/*"
{code}
This adds way way way more than you need and should be specifically avoided in
trunk. So don't do that. Besides, if hadoop-aws is enabled as a supported tool,
the AWS bits are already present in the classpath. If you want to forcibly
load them regardless of the HADOOP_OPTIONAL_TOOLS setting, then call
_hadoop-aws_hadoop_classpath directly. If the hadoop-aws support has been
already added, hadoop_add_classpath will skip adding the jars a second time.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13597) Switch KMS from Tomcat to Jetty
[
https://issues.apache.org/jira/browse/HADOOP-13597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704129#comment-15704129
]
John Zhuge edited comment on HADOOP-13597 at 11/29/16 6:08 PM:
---
Patch 001
- KMSHttpServer based on HttpServer2
- Redirect MiniKMS to KMSHttpServer
- Add kms-default.xml
- Add Jetty properties including SSL properties
- Convert hadoop-kms from war to jar
- Rewrite kms.sh to use Hadoop shell script framework
- Obsolete HTTP admin port for the Tomcat Manager GUI which does not seem to
work anyway
- Obsolete {{kms.sh version}} that prints Tomcat version
TESTING DONE
- All hadoop-kms unit tests. MiniKMS equals full KMS.
- Non-secure REST APIs
- Non-secure “hadoop key” commands
- SSL REST APIs
- kms.sh run/start/stop/status
- JMX works
- /logs works
TODO
- Set HTTP request header size by env KMS_MAX_HTTP_HEADER_SIZE
- Add static web content /index.html
- More ad-hoc testing
- Integration testing
- Update docs: index.md.vm
TODO in new JIRAs:
- Integrate with Hadoop SSL server configuration
- Full SSL server configuration:
includeProtocols/excludeProtocols/includeCipherSuites/excludeCipherSuites, etc.
- Design common Http server configuration. Common properties in
“-site.xml” with config prefix, e.g., “hadoop.kms.”.
- Design HttpServer2 configuration-based builder
- Share web apps code in Common, HDFS, and YARN
My private branch: https://github.com/jzhuge/hadoop/tree/HADOOP-13597.001
was (Author: jzhuge):
Patch 001
- KMSHttpServer based on HttpServer2
- Redirect MiniKMS to KMSHttpServer
- Add kms-default.xml
- Add Jetty properties including SSL properties
- Convert hadoop-kms from war to jar
- Rewrite kms.sh to use Hadoop shell script framework
- Obsolete HTTP admin port for the Tomcat Manager GUI which does not seem to
work anyway
- Obsolete {{kms.sh version}} that prints Tomcat version
TESTING DONE
- All hadoop-kms unit tests which exercise the full KMS instead of MiniKMS
- Non-secure REST APIs
- Non-secure “hadoop key” commands
- SSL REST APIs
- kms.sh run/start/stop/status
TODO
- Set HTTP request header size by env KMS_MAX_HTTP_HEADER_SIZE
- Add static web content /index.html
- More ad-hoc testing
- Integration testing
- Update docs: index.md.vm
TODO in new JIRAs:
- Integrate with Hadoop SSL server configuration
- Full SSL server configuration:
includeProtocols/excludeProtocols/includeCipherSuites/excludeCipherSuites, etc.
- Design common Http server configuration. Common properties in
“-site.xml” with config prefix, e.g., “hadoop.kms.”.
- Design HttpServer2 configuration-based builder
- JMX not working, existing issue
- Share web apps code in Common, HDFS, and YARN
My private branch: https://github.com/jzhuge/hadoop/tree/HADOOP-13597.001
> Switch KMS from Tomcat to Jetty
> ---
>
> Key: HADOOP-13597
> URL: https://issues.apache.org/jira/browse/HADOOP-13597
> Project: Hadoop Common
> Issue Type: New Feature
> Components: kms
>Affects Versions: 2.6.0
>Reporter: John Zhuge
>Assignee: John Zhuge
> Attachments: HADOOP-13597.001.patch
>
>
> The Tomcat 6 we are using will reach EOL at the end of 2017. While there are
> other good options, I would propose switching to {{Jetty 9}} for the
> following reasons:
> * Easier migration. Both Tomcat and Jetty are based on {{Servlet
> Containers}}, so we don't have change client code that much. It would require
> more work to switch to {{JAX-RS}}.
> * Well established.
> * Good performance and scalability.
> Other alternatives:
> * Jersey + Grizzly
> * Tomcat 8
> Your opinions will be greatly appreciated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706026#comment-15706026 ] Arun Suresh commented on HADOOP-13836: -- Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certification authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706026#comment-15706026 ] Arun Suresh edited comment on HADOOP-13836 at 11/29/16 6:02 PM: Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certificate authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) was (Author: asuresh): Thanks for pointing out IPv6 [~steve_l] I assume it should just work, considering this comes into play only at Socket creation. My understanding is that SSL/TLS certification authentication is based on the DNS names of the entities involved, so again, I assume it should just work, if DNS resolution works correctly. But yes, we will try to verify it. My understanding is that the HADOOP-11890 branch had most of the IPv6 based changes. If this branch is uptodate, [~kartheek], we can maybe test it against that branch. Any more pointers to verifying if this would work with IPv6 is welcome. (cc: [~eclark], [~nkedel]) > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13600: Assignee: Sahil Takiar (was: Steve Loughran) > S3a rename() to copy files in a directory in parallel > - > > Key: HADOOP-13600 > URL: https://issues.apache.org/jira/browse/HADOOP-13600 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Sahil Takiar > > Currently a directory rename does a one-by-one copy, making the request > O(files * data). If the copy operations were launched in parallel, the > duration of the copy may be reducable to the duration of the longest copy. > For a directory with many files, this will be significant -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13845) s3a to instrument duration of HTTP calls
[ https://issues.apache.org/jira/browse/HADOOP-13845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13845: Priority: Minor (was: Major) > s3a to instrument duration of HTTP calls > > > Key: HADOOP-13845 > URL: https://issues.apache.org/jira/browse/HADOOP-13845 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.9.0 >Reporter: Steve Loughran >Priority: Minor > > HADOOP-13844 proposes pulling out the swift duration classes for reuse; this > patch proposes instrumenting s3a with it. > One interesting question: what to do with the values. For now, they could > just be printed, but it might be interesting to include in FS stats collected > at the end of a run. However, those are all assumed to be simple counters > where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13845) s3a to instrument duration of HTTP calls
Steve Loughran created HADOOP-13845: --- Summary: s3a to instrument duration of HTTP calls Key: HADOOP-13845 URL: https://issues.apache.org/jira/browse/HADOOP-13845 Project: Hadoop Common Issue Type: Sub-task Components: fs/s3 Affects Versions: 2.9.0 Reporter: Steve Loughran HADOOP-13844 proposes pulling out the swift duration classes for reuse; this patch proposes instrumenting s3a with it. One interesting question: what to do with the values. For now, they could just be printed, but it might be interesting to include in FS stats collected at the end of a run. However, those are all assumed to be simple counters where merging is a matter of addition. These are more metrics -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13204: Comment: was deleted (was: Patch 008; in sync with branch-2) > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13204) Über-jira: S3a phase III: scale and tuning
[ https://issues.apache.org/jira/browse/HADOOP-13204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-13204: Attachment: (was: HADOOP-13614-branch-2-008.patch) > Über-jira: S3a phase III: scale and tuning > -- > > Key: HADOOP-13204 > URL: https://issues.apache.org/jira/browse/HADOOP-13204 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > S3A Phase III work; post 2.8. > Areas could include > * customisation > * performance enhancement > * management and metrics > * error/failure handling > And of course any bugs which surface -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13844) Move swift duration stats tracking to hadoop common utils
Steve Loughran created HADOOP-13844: --- Summary: Move swift duration stats tracking to hadoop common utils Key: HADOOP-13844 URL: https://issues.apache.org/jira/browse/HADOOP-13844 Project: Hadoop Common Issue Type: Improvement Components: fs/azure, fs/oss, fs/s3, fs/swift Affects Versions: 2.9.0 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Swift FS has some simple duration counting classes which are used to track HTTP performance, with incremental creation of mean, min, max and variance. These are not moving averages, simply ongoing over time. They've proved useful in detecting which operations are slowest against objects stores. (it turned out to be delete throttling in test teardown BTW, showing the value of the instrumentation) By moving them into utils, all the object stores can build up some stats on cost of HTTP operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705730#comment-15705730 ] Steve Loughran commented on HADOOP-13836: - This is all going to work once the IPV6 patch is merged in, right? > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705693#comment-15705693 ] kartheek muthyala commented on HADOOP-13836: [~antkou], I liked your suggestion of using "hadoop.rpc.socket.factory.class.default" to specify our own SSLSocketFactory class on the Client side. For this I just have to move the existing SSL socket creation related changes from Client.java to a separate file. Will consider this in my next patch. We have considered org.apache.hadoop.security.ssl.SSLFactory for creating SSLEngine on Server. But it doesn't has SSLSocketChannel on top of SSLEngine, like niossl library does. Having this support would make us stick to the existing SocketChannel Server design. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13843) S3Guard, MetadataStore to support atomic create(path, overwrite=false)
Steve Loughran created HADOOP-13843:
---
Summary: S3Guard, MetadataStore to support atomic create(path,
overwrite=false)
Key: HADOOP-13843
URL: https://issues.apache.org/jira/browse/HADOOP-13843
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.0.0-alpha2
Reporter: Steve Loughran
Support atomically enforced file creation. Current s3a can do a check in
create() and fail if there is something there, but a new entry only gets
created at the end of the PUT; during the entire interval between that check
and the close() of the stream, there's nothing to stop other callers creating
an object.
Proposed: s3afs can do a check + create a 0 byte file at the path; that'd need
some {{putNoOverwrite(DirListingMetadata)}} call in MetadataStore, followed by
a PUT of an 0-byte file to S3. That will increase cost of file creation, though
at least with the MD store, the cost of the initial getFileStatus() check is
down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13786) add output committer which uses s3guard for consistent O(1) commits to S3
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705650#comment-15705650 ] Steve Loughran commented on HADOOP-13786: - Looking at the atomic commit problem, I'm starting to worry that I may need some directory leasing, which would imply a lock field in the table. Just warning you there. The problem is that while a rename is in progress, you wan't to stop anything trying to PUT data anywhere under the two paths being worked on. > add output committer which uses s3guard for consistent O(1) commits to S3 > - > > Key: HADOOP-13786 > URL: https://issues.apache.org/jira/browse/HADOOP-13786 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.0.0-alpha2 >Reporter: Steve Loughran >Assignee: Steve Loughran > > A goal of this code is "support O(1) commits to S3 repositories in the > presence of failures". Implement it, including whatever is needed to > demonstrate the correctness of the algorithm. (that is, assuming that s3guard > provides a consistent view of the presence/absence of blobs, show that we can > commit directly). > I consider ourselves free to expose the blobstore-ness of the s3 output > streams (ie. not visible until the close()), if we need to use that to allow > us to abort commit operations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705643#comment-15705643
]
Steve Loughran commented on HADOOP-13650:
-
* {{S3GuardTool.main}} needs to catch, log all exceptions. Ideally return a
different exit code for an exception than from a usage command; this makes
testing/diags easier.
* be good to have ~aw to review the bash scripts. Are these testable? At the
very least, something that goes as far as a usage command in the java code can
do basic installation testing.
* we have the option of JCommander here for arg parsing. just a thought.
* might be good to have the option of printing the diff out in a way that's
easy to parse downstream. Is the current output machine parseable?
* For the comparison, a {{listFiles(recursive=true)}} is much faster to list s3
buckets...but I'm not sure how well it will work here, because the iterator
returned from the metastore may return differently ordered results from the FS;
you don't want to build up large structures in memory. Even so, I'm not sure
how well the existing code can handle a path with a few million blobs directly
under a path. some s3 buckets do have this, apparently.
* Maybe an operation to verify that the metastore is in sync with s3, return a
specific error code to indicate this is not the case. That's something you can
use at installation, preflight, etc.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13650) S3Guard: Provide command line tools to manipulate metadata store.
[
https://issues.apache.org/jira/browse/HADOOP-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705643#comment-15705643
]
Steve Loughran edited comment on HADOOP-13650 at 11/29/16 3:42 PM:
---
* {{S3GuardTool.main}} needs to catch, log all exceptions. Ideally return a
different exit code for an exception than from a usage command; this makes
testing/diags easier.
* be good to have [~aw] to review the bash scripts. Are these testable? At the
very least, something that goes as far as a usage command in the java code can
do basic installation testing.
* we have the option of JCommander here for arg parsing. just a thought.
* might be good to have the option of printing the diff out in a way that's
easy to parse downstream. Is the current output machine parseable?
* For the comparison, a {{listFiles(recursive=true)}} is much faster to list s3
buckets...but I'm not sure how well it will work here, because the iterator
returned from the metastore may return differently ordered results from the FS;
you don't want to build up large structures in memory. Even so, I'm not sure
how well the existing code can handle a path with a few million blobs directly
under a path. some s3 buckets do have this, apparently.
* Maybe an operation to verify that the metastore is in sync with s3, return a
specific error code to indicate this is not the case. That's something you can
use at installation, preflight, etc.
was (Author: ste...@apache.org):
* {{S3GuardTool.main}} needs to catch, log all exceptions. Ideally return a
different exit code for an exception than from a usage command; this makes
testing/diags easier.
* be good to have ~aw to review the bash scripts. Are these testable? At the
very least, something that goes as far as a usage command in the java code can
do basic installation testing.
* we have the option of JCommander here for arg parsing. just a thought.
* might be good to have the option of printing the diff out in a way that's
easy to parse downstream. Is the current output machine parseable?
* For the comparison, a {{listFiles(recursive=true)}} is much faster to list s3
buckets...but I'm not sure how well it will work here, because the iterator
returned from the metastore may return differently ordered results from the FS;
you don't want to build up large structures in memory. Even so, I'm not sure
how well the existing code can handle a path with a few million blobs directly
under a path. some s3 buckets do have this, apparently.
* Maybe an operation to verify that the metastore is in sync with s3, return a
specific error code to indicate this is not the case. That's something you can
use at installation, preflight, etc.
> S3Guard: Provide command line tools to manipulate metadata store.
> -
>
> Key: HADOOP-13650
> URL: https://issues.apache.org/jira/browse/HADOOP-13650
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
> Attachments: HADOOP-13650-HADOOP-13345.000.patch,
> HADOOP-13650-HADOOP-13345.001.patch, HADOOP-13650-HADOOP-13345.002.patch
>
>
> Similar systems like EMRFS has the CLI tools to manipulate the metadata
> store, i.e., create or delete metadata store, or {{import}}, {{sync}} the
> file metadata between metadata store and S3.
> http://docs.aws.amazon.com//ElasticMapReduce/latest/ReleaseGuide/emrfs-cli-reference.html
> S3Guard should offer similar functionality.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705634#comment-15705634 ] John Zhuge commented on HADOOP-13812: - [~ajisakaa] So many -1's with 2.6 patch? > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13421) Switch to v2 of the S3 List Objects API in S3A
[
https://issues.apache.org/jira/browse/HADOOP-13421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705631#comment-15705631
]
Thomas Demoor commented on HADOOP-13421:
Thanks for reaching out [~ste...@apache.org] ([~PieterReuse] discussed with
me).
This API call is quite recent (May 4th 2016) so there are a lot of legacy
systems which will not support this. Our latest release does but I think if we
don't make this optional the majority of other s3 endpoints will brake.
Making this optional seems relatively low cost. I quickly checked and we don't
seem to use the {{marker}} functionality in the V1 implementation which is
replaced by {{start-after}} in V2.
> Switch to v2 of the S3 List Objects API in S3A
> --
>
> Key: HADOOP-13421
> URL: https://issues.apache.org/jira/browse/HADOOP-13421
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0
>Reporter: Steven K. Wong
>Priority: Minor
>
> Unlike [version
> 1|http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html] of the
> S3 List Objects API, [version
> 2|http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html] by
> default does not fetch object owner information, which S3A doesn't need
> anyway. By switching to v2, there will be less data to transfer/process.
> Also, it should be more robust when listing a versioned bucket with "a large
> number of delete markers" ([according to
> AWS|https://aws.amazon.com/releasenotes/Java/0735652458007581]).
> Methods in S3AFileSystem that use this API include:
> * getFileStatus(Path)
> * innerDelete(Path, boolean)
> * innerListStatus(Path)
> * innerRename(Path, Path)
> Requires AWS SDK 1.10.75 or later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated HADOOP-13812: Release Note: Tomcat 6.0.46 starts to filter weak ciphers. Some old SSL clients may be affected. It is recommended to upgrade the SSL client. Run the SSL client against https://www.howsmyssl.com/a/check to find out its TLS version and cipher suites. (was: Tomcat 6.0.46 starts to filter weak ciphers. Some old SSL clients may be affected. It is recommended to upgrade the SSL client.) > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705614#comment-15705614 ] kartheek muthyala commented on HADOOP-13836: [~drankye] , very good questions. Here are some of my responses. Correct me if I am wrong. I will soon post a design doc with all these details What's the scenarios, requirements and use cases you have in mind for this support (other than Kerberos)? - Avoiding man in the middle attacks, through proper SSL Connection handshake before even exchanging the data. - Better encryption over the wire. - Ability to get trusted third party validation through Versign, goDaddy etc, which improves industrial adoption for sensitive data exchange. - We can extend the same cipher suite to encrypt data on flight and rest. What interfaces will be taken care of by this: RPC/commands, REST, web, JDBC and etc. - For now we are supporting interfaces that derive from the hadoop.ipc.Server and hadoop.ipc.Client classes. So, primarily RPC. How authentication will be considered? Still simple or some mechanisms over SSL/TLS? - Today we enabled client to authenticate with the servers on connection. So configuring keystore is a must on the server. And with a configured KeyManager we can decide on what authentication credentials should be sent to the remote host for authentication during SSL handshake. How would you manage credentials (X.509 certificates) for Hadoop services and maybe clients? - The current work requires both server and client to be installed with keystore and truststore, and configured through ssl-client.xml and ssl-server.xml. What's the exact SSL/TLS versions to support and how to configure such with the cipher suite options? - Currently we are supporting TLSv1.2 as a default. Because the cipher suite hasn't changed between TLSv1 and TLSv1.2, TLSv1 also should be supported. Given that the SSLContext varies for different versions of SSL/TLS, we can provide an interface for deriving this SSLContext depending upon the version of TLS/SSL configured. > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13589) S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
[
https://issues.apache.org/jira/browse/HADOOP-13589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705598#comment-15705598
]
Steve Loughran commented on HADOOP-13589:
-
now we've got multiple back ends (and dynamo soon), life is complex. I'd like
to have the option to mandate s3guard, which if set goes to local
(authoritative = ? ), but allow us to switch to dynamodb as well.
{{S3ATestUtils.getTestProperty()}} will grab a property for a test, giving a
sysprop which != "unset" the ability to override anything in the XML test
configurations. This would let you
# have a property {{-Ds3guard}}
# which will enable an {{s3guard}} profile to set what we want
if we modify the surefire/failsafe profiles to pass down a set of system
properties, which are all set to {{unset}} by default then any of those set in
the s3guard profile, or set on the maven command line will be picked up and
override any in the XML files.
The other thing to consider is junit profiles. HADOOP-13834 covers that idea as
a way of enabling scale tests, rather than the contrived bits in
{{AbstractS3aScaleTests}}. I haven't got that working, and haven't really put
in the time to understand why —it felt like one of those interesting wastes of
large amounts of time. A thought though.
> S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
> ---
>
> Key: HADOOP-13589
> URL: https://issues.apache.org/jira/browse/HADOOP-13589
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Reporter: Chris Nauroth
>Assignee: Steve Loughran
>
> With S3Guard enabled, S3A must continue to be functionally correct. The best
> way to verify this is to execute our existing S3A integration tests in a mode
> with S3A enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705577#comment-15705577 ] John Zhuge commented on HADOOP-13812: - Thanks [~ajisakaa] for the review and commit. > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13836) Securing Hadoop RPC using SSL
[ https://issues.apache.org/jira/browse/HADOOP-13836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-13836: - Assignee: kartheek muthyala > Securing Hadoop RPC using SSL > - > > Key: HADOOP-13836 > URL: https://issues.apache.org/jira/browse/HADOOP-13836 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Reporter: kartheek muthyala >Assignee: kartheek muthyala > Attachments: HADOOP-13836.patch > > > Today, RPC connections in Hadoop are encrypted using Simple Authentication & > Security Layer (SASL), with the Kerberos ticket based authentication or > Digest-md5 checksum based authentication protocols. This proposal is about > enhancing this cipher suite with SSL/TLS based encryption and authentication. > SSL/TLS is a proposed Internet Engineering Task Force (IETF) standard, that > provides data security and integrity across two different end points in a > network. This protocol has made its way to a number of applications such as > web browsing, email, internet faxing, messaging, VOIP etc. And supporting > this cipher suite at the core of Hadoop would give a good synergy with the > applications on top and also bolster industry adoption of Hadoop. > The Server and Client code in Hadoop IPC should support the following modes > of communication > 1.Plain > 2. SASL encryption with an underlying authentication > 3. SSL based encryption and authentication (x509 certificate) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13332) Remove jackson 1.9.13 and switch all jackson code to 2.x code line
[ https://issues.apache.org/jira/browse/HADOOP-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705399#comment-15705399 ] Akira Ajisaka commented on HADOOP-13332: bq. What uses jersey-json? ATS rest client? Used by JAXBContextResolver.java in the following modules: * hadoop-yarn-server-nodemanager * hadoop-yarn-server-resourcemanager * hadoop-yarn-server-applicationhistoryservice * hadoop-yarn-common * hadoop-mapreduce-client-app * hadoop-mapreduce-client-hs > Remove jackson 1.9.13 and switch all jackson code to 2.x code line > -- > > Key: HADOOP-13332 > URL: https://issues.apache.org/jira/browse/HADOOP-13332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: PJ Fanning >Assignee: Akira Ajisaka > Attachments: HADOOP-13332.00.patch, HADOOP-13332.01.patch, > HADOOP-13332.02.patch, HADOOP-13332.03.patch > > > This jackson 1.9 code line is no longer maintained. Upgrade > Most changes from jackson 1.9 to 2.x just involve changing the package name. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13332) Remove jackson 1.9.13 and switch all jackson code to 2.x code line
[ https://issues.apache.org/jira/browse/HADOOP-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705250#comment-15705250 ] Steve Loughran commented on HADOOP-13332: - hbase is only used for ATS -hbase tests, AFAIK. What uses jersey-json? ATS rest client? > Remove jackson 1.9.13 and switch all jackson code to 2.x code line > -- > > Key: HADOOP-13332 > URL: https://issues.apache.org/jira/browse/HADOOP-13332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: PJ Fanning >Assignee: Akira Ajisaka > Attachments: HADOOP-13332.00.patch, HADOOP-13332.01.patch, > HADOOP-13332.02.patch, HADOOP-13332.03.patch > > > This jackson 1.9 code line is no longer maintained. Upgrade > Most changes from jackson 1.9 to 2.x just involve changing the package name. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13842) Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
[
https://issues.apache.org/jira/browse/HADOOP-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705246#comment-15705246
]
Hadoop QA commented on HADOOP-13842:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
11s{color} | {color:green} hadoop-maven-plugins in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | HADOOP-13842 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12840867/HADOOP-13842.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 74a3b1c9a081 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 00096dc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11160/testReport/ |
| modules | C: hadoop-maven-plugins U: hadoop-maven-plugins |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/11160/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
> -
>
> Key: HADOOP-13842
> URL: https://issues.apache.org/jira/browse/HADOOP-13842
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: build
>Reporter: Akira
[jira] [Commented] (HADOOP-13600) S3a rename() to copy files in a directory in parallel
[ https://issues.apache.org/jira/browse/HADOOP-13600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705228#comment-15705228 ] Steve Loughran commented on HADOOP-13600: - yeah, looking at my patch I'm not sure if there's much of it worth preserving. I'd realised it had got over complex. How about you sync your patch up with HADOOP-13823 and we start from there. Elsewhere (HADOOP-13786) I'm doing a special committer for s3, which is going to directly call innerRename() & so relay failures up immediately. Failures on renames is going to be fun; # I can think of a test for that but it will be fairly brittle: create two large files, have a thread size of 1 for the copies, delete the 2nd while the first copy is in progress but after the listing has started. Or: start playing with mocks via the plug in AWS client code; have something which explicitly triggers failures on copy requests. # I do want at least the first copy failure to be propagated up to the caller; usually it is that first failure which is most interesting. > S3a rename() to copy files in a directory in parallel > - > > Key: HADOOP-13600 > URL: https://issues.apache.org/jira/browse/HADOOP-13600 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.7.3 >Reporter: Steve Loughran >Assignee: Steve Loughran > > Currently a directory rename does a one-by-one copy, making the request > O(files * data). If the copy operations were launched in parallel, the > duration of the copy may be reducable to the duration of the longest copy. > For a directory with many files, this will be significant -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13332) Remove jackson 1.9.13 and switch all jackson code to 2.x code line
[
https://issues.apache.org/jira/browse/HADOOP-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705216#comment-15705216
]
Akira Ajisaka commented on HADOOP-13332:
Filed HADOOP-13842 and attached a patch there.
Now jersey-json, avro, hbase-server, and hbase-client depend on jackson 1.9.x.
We cannot remove the setting in hadoop-project pom.xml.
{noformat:title=mvn dependency:tree}
[INFO] +- com.sun.jersey:jersey-json:jar:1.19:compile
[INFO] | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | \- javax.xml.bind:jaxb-api:jar:2.2.11:compile (version managed
from 2.2.2)
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile (version
managed from 1.9.2)
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
(version managed from 1.9.2)
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile (version
managed from 1.9.2)
[INFO] | \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile (version
managed from 1.9.2)
{noformat}
> Remove jackson 1.9.13 and switch all jackson code to 2.x code line
> --
>
> Key: HADOOP-13332
> URL: https://issues.apache.org/jira/browse/HADOOP-13332
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.8.0
>Reporter: PJ Fanning
>Assignee: Akira Ajisaka
> Attachments: HADOOP-13332.00.patch, HADOOP-13332.01.patch,
> HADOOP-13332.02.patch, HADOOP-13332.03.patch
>
>
> This jackson 1.9 code line is no longer maintained. Upgrade
> Most changes from jackson 1.9 to 2.x just involve changing the package name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13842) Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
[ https://issues.apache.org/jira/browse/HADOOP-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13842: --- Priority: Minor (was: Major) > Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins > - > > Key: HADOOP-13842 > URL: https://issues.apache.org/jira/browse/HADOOP-13842 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Minor > Attachments: HADOOP-13842.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13842) Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
[ https://issues.apache.org/jira/browse/HADOOP-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13842: --- Status: Patch Available (was: Open) > Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins > - > > Key: HADOOP-13842 > URL: https://issues.apache.org/jira/browse/HADOOP-13842 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HADOOP-13842.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13842) Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
[ https://issues.apache.org/jira/browse/HADOOP-13842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13842: --- Attachment: HADOOP-13842.01.patch > Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins > - > > Key: HADOOP-13842 > URL: https://issues.apache.org/jira/browse/HADOOP-13842 > Project: Hadoop Common > Issue Type: Sub-task > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka > Attachments: HADOOP-13842.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13842) Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins
Akira Ajisaka created HADOOP-13842: -- Summary: Update jackson from 1.9.13 to 2.x in hadoop-maven-plugins Key: HADOOP-13842 URL: https://issues.apache.org/jira/browse/HADOOP-13842 Project: Hadoop Common Issue Type: Sub-task Reporter: Akira Ajisaka Assignee: Akira Ajisaka -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-13841) S3a auth exception to link to a wiki page on the problem
Steve Loughran created HADOOP-13841:
---
Summary: S3a auth exception to link to a wiki page on the problem
Key: HADOOP-13841
URL: https://issues.apache.org/jira/browse/HADOOP-13841
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.8.0
Reporter: Steve Loughran
Priority: Minor
Currently when there's an auth problem, the user gets a stack trace followed by
"Invalid arguments: Unable to load AWS credentials from any provider in the
chain"
this is explained in the AWS troubleshooting docs, but unless the user knows
about that, they aren't going to make much progress. Better to have
{{S3AUtils.translateException()}} point to a wiki page on the topic
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[
https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705118#comment-15705118
]
Hadoop QA commented on HADOOP-13812:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
48s{color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
42s{color} | {color:red} root in branch-2.6 failed with JDK v1.8.0_111. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
48s{color} | {color:red} root in branch-2.6 failed with JDK v1.7.0_121. {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} branch-2.6 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
53s{color} | {color:green} branch-2.6 passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs-httpfs in branch-2.6 failed with JDK
v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} branch-2.6 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
34s{color} | {color:red} root in the patch failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 34s{color}
| {color:red} root in the patch failed with JDK v1.8.0_111. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
45s{color} | {color:red} root in the patch failed with JDK v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 45s{color}
| {color:red} root in the patch failed with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2048 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
55s{color} | {color:red} The patch 35 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs-httpfs in the patch failed with JDK
v1.8.0_111. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
8s{color} | {color:green} hadoop-project in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
7s{color} | {color:green} hadoop-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
22s{color} | {color:green} hadoop-kms in the patch passed with JDK v1.7.0_121.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed with JDK
v1.7.0_121. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
32s{color} | {color:red} The patch generated 164 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HADOOP-13332) Remove jackson 1.9.13 and switch all jackson code to 2.x code line
[ https://issues.apache.org/jira/browse/HADOOP-13332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705112#comment-15705112 ] Steve Loughran commented on HADOOP-13332: - and there was me thinking you were done... > Remove jackson 1.9.13 and switch all jackson code to 2.x code line > -- > > Key: HADOOP-13332 > URL: https://issues.apache.org/jira/browse/HADOOP-13332 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 2.8.0 >Reporter: PJ Fanning >Assignee: Akira Ajisaka > Attachments: HADOOP-13332.00.patch, HADOOP-13332.01.patch, > HADOOP-13332.02.patch, HADOOP-13332.03.patch > > > This jackson 1.9 code line is no longer maintained. Upgrade > Most changes from jackson 1.9 to 2.x just involve changing the package name. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13840) Implement getUsed() for ViewFileSystem
[
https://issues.apache.org/jira/browse/HADOOP-13840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705110#comment-15705110
]
Steve Loughran commented on HADOOP-13840:
-
can you switch to {{assertEquals()}} for the equality test in the assertion; it
provides meaningful details on a failure *provided the expected value comes
first(
{code}
assertEquals("space used", usedSpceByPathViaTargetFs ,usedSpcByPathViaViewFs)
{code}
> Implement getUsed() for ViewFileSystem
> --
>
> Key: HADOOP-13840
> URL: https://issues.apache.org/jira/browse/HADOOP-13840
> Project: Hadoop Common
> Issue Type: Task
> Components: viewfs
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Attachments: HADOOP-13840.01.patch
>
>
> ViewFileSystem doesn't override {{FileSystem#getUSed()}}. So, when file
> system used space is queried for slash root "/" paths, the default
> implementations tries to run the {{getContentSummary}} which crashes on
> seeing unexpected mount points under slash.
> ViewFileSystem#getUsed() is not expected to collate all the space used from
> all the mount points configured under "/". Proposal is to avoid invoking
> FileSystem#getUsed() and throw NotInMountPointException until LinkMergeSlash
> is supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13826) S3A Deadlock in multipart copy due to thread pool limits.
[
https://issues.apache.org/jira/browse/HADOOP-13826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705101#comment-15705101
]
Steve Loughran commented on HADOOP-13826:
-
they're kept consistent for a reason, not just because it simplifies
configuration in general.
S3 stores separate parts in separate files; you should get better performance
when reading parts separately; that is, for max speed you should set s3a block
size == upload partition size. By doing a copy with copy partition size ==
upload part size, we hope to preserve that performance on later reads. Who
knows, maybe it will even help copy performance.
what would be ideal would be to know the part size of an object; HADOOP-13261
proposed adding a custom header for this. However, time spent looking at split
calculation performance has convinced me that a new header would be useless
there; the overhead of querying the objects makes it too expensive. We could
start uploading it though, and maybe use it for a copy. still expensive though;
a 400mS HEAD would be about 2MB of copy bandwidth based on my (ad-hoc)
measurements of copy B/W of 6 MB/s
> S3A Deadlock in multipart copy due to thread pool limits.
> -
>
> Key: HADOOP-13826
> URL: https://issues.apache.org/jira/browse/HADOOP-13826
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 2.7.3
>Reporter: Sean Mackrory
> Attachments: HADOOP-13826.001.patch, HADOOP-13826.002.patch
>
>
> In testing HIVE-15093 we have encountered deadlocks in the s3a connector. The
> TransferManager javadocs
> (http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/transfer/TransferManager.html)
> explain how this is possible:
> {quote}It is not recommended to use a single threaded executor or a thread
> pool with a bounded work queue as control tasks may submit subtasks that
> can't complete until all sub tasks complete. Using an incorrectly configured
> thread pool may cause a deadlock (I.E. the work queue is filled with control
> tasks that can't finish until subtasks complete but subtasks can't execute
> because the queue is filled).{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13261) save partition split size on multipart uploads
[ https://issues.apache.org/jira/browse/HADOOP-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705095#comment-15705095 ] Steve Loughran commented on HADOOP-13261: - it'd be expensive to query though, as you do not get this information back on a LIST; you'd need to do a HEAD on the file —way too expensive to use in split calculation. Where it could be used is in copy and perhaps an s3-specific distcp, where the partition size could be propagated > save partition split size on multipart uploads > -- > > Key: HADOOP-13261 > URL: https://issues.apache.org/jira/browse/HADOOP-13261 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Priority: Minor > > On multipart uploads, save the split size as a metadata value. This would > allow split calculation optimized for the partitions to be performed in some > bulk operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705056#comment-15705056
]
Steve Loughran edited comment on HADOOP-13811 at 11/29/16 11:45 AM:
you are getting an exception on line 248 of S3a, where it's calling {{partSize
= conf.getLong(MULTIPART_SIZE, DEFAULT_MULTIPART_SIZE)}}.
Now, that line only exists on branch-2.7.x; it's not on the branch-2 line,
where we use {{getLongBytes()}} to support byte unit measurement s (M, K, P, T,
G). Which means that somehow you are getting to 2.7 binaries in your jobs.
You could make that error move off that line by changing the option
{{fs.s3a.multipart.size}} to {{104857600}}. But you've still got that mix of
binaries there, and that means your problem isn't going to go away.
was (Author: ste...@apache.org):
you are getting an exception on line 248 of S3a, where it's calling {{partSize
= conf.getLong(MULTIPART_SIZE, DEFAULT_MULTIPART_SIZE)}}.
Now, that line only exists on branch-2.7.x; it's not on the branch-2 line,
where we use getLongBytes() to support byte unit measurement s (M, K, P, T, G).
Which means that somehow you are getting to 2.7 binaries in your jobs.
You could make that error move off that line by changing the option
{{fs.s3a.multipart.size}} to {{104857600}}. But you've still got that mix of
binaries there, and that means your problem isn't going to go away.
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13811) s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to sanitize XML document destined for handler class
[
https://issues.apache.org/jira/browse/HADOOP-13811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705056#comment-15705056
]
Steve Loughran commented on HADOOP-13811:
-
you are getting an exception on line 248 of S3a, where it's calling {{partSize
= conf.getLong(MULTIPART_SIZE, DEFAULT_MULTIPART_SIZE)}}.
Now, that line only exists on branch-2.7.x; it's not on the branch-2 line,
where we use getLongBytes() to support byte unit measurement s (M, K, P, T, G).
Which means that somehow you are getting to 2.7 binaries in your jobs.
You could make that error move off that line by changing the option
{{fs.s3a.multipart.size}} to {{104857600}}. But you've still got that mix of
binaries there, and that means your problem isn't going to go away.
> s3a: getFileStatus fails with com.amazonaws.AmazonClientException: Failed to
> sanitize XML document destined for handler class
> -
>
> Key: HADOOP-13811
> URL: https://issues.apache.org/jira/browse/HADOOP-13811
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 2.8.0, 2.7.3
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Sometimes, occasionally, getFileStatus() fails with a stack trace starting
> with {{com.amazonaws.AmazonClientException: Failed to sanitize XML document
> destined for handler class}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13812: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed (was: Incompatible change) Fix Version/s: 2.6.6 3.0.0-alpha2 2.7.4 2.8.0 Status: Resolved (was: Patch Available) Committed this to branch-2.6 as well. Thanks [~jzhuge] for the contribution! > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6 > > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka updated HADOOP-13812: --- Attachment: HADOOP-13812.branch-2.6.patch Committed this to trunk, branch-2, branch-2.8, branch-2.7. Now cherry-picking to branch-2.6. I had to rebase the patch for branch-2.6, so uploading this. > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Attachments: HADOOP-13812.001.patch, HADOOP-13812.branch-2.6.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13794) JSON.org license is now CatX
[
https://issues.apache.org/jira/browse/HADOOP-13794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704983#comment-15704983
]
Steve Loughran commented on HADOOP-13794:
-
Andrew Wang has pointed to:
https://lists.apache.org/thread.html/bb18f942ce7eb83c11438303c818b885810fb76385979490366720d5@%3Clegal-discuss.apache.org%3E
bq. If you have been using it, and have done so in a *release*, AND there has
been NO pushback from your community/eco-system, you have a temporary exclusion
from the Cat-X classification thru April 30, 2017. At that point in time, ANY
and ALL usage of these JSON licensed artifacts are DISALLOWED. You must either
find a suitably licensed replacement, or do without. There will be NO
exceptions.
This means that Hadoop 2.6-2.8 can continue to ship the older libs —with that
disclaimer; with the jackson/aws SDK updates there's no issue in 2.9+ trunk.
This gives us some headroom to come up with some shading.
What it doesn't do is let projects downstream, which weren't bundling the AWS
SDK, to start doing so.
> JSON.org license is now CatX
>
>
> Key: HADOOP-13794
> URL: https://issues.apache.org/jira/browse/HADOOP-13794
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.8.0, 2.7.4, 3.0.0-alpha2, 2.6.6
>Reporter: Sean Busbey
>Priority: Blocker
>
> per [update resolved legal|http://www.apache.org/legal/resolved.html#json]:
> {quote}
> CAN APACHE PRODUCTS INCLUDE WORKS LICENSED UNDER THE JSON LICENSE?
> No. As of 2016-11-03 this has been moved to the 'Category X' license list.
> Prior to this, use of the JSON Java library was allowed. See Debian's page
> for a list of alternatives.
> {quote}
> We have a test-time transitive dependency on the {{org.json:json}} artifact
> in trunk and branch-2. AFAICT, this test time dependency doesn't get exposed
> to downstream at all (I checked assemblies and test-jar artifacts we publish
> to maven), so it can be removed or kept at our leisure. keeping it risks it
> being promoted out of test scope by maven without us noticing. We might be
> able to add an enforcer rule to check for this.
> We also distribute it in bundled form through our use of the AWS Java SDK
> artifacts in trunk and branch-2. Looking at the github project, [their
> dependency on JSON.org was removed in
> 1.11|https://github.com/aws/aws-sdk-java/pull/417], so if we upgrade to
> 1.11.0+ we should be good to go. (this might be hard in branch-2.6 and
> branch-2.7 where we're on 1.7.4)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-13812) Upgrade Tomcat to 6.0.48
[ https://issues.apache.org/jira/browse/HADOOP-13812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15704967#comment-15704967 ] Hudson commented on HADOOP-13812: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10905 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10905/]) HADOOP-13812. Upgrade Tomcat to 6.0.48. Contributed by John Zhuge. (aajisaka: rev 00096dcc0c23af38e697cd7c6c29e3f00d744e54) * (edit) hadoop-common-project/hadoop-kms/src/site/markdown/index.md.vm * (edit) hadoop-hdfs-project/hadoop-hdfs-httpfs/src/site/markdown/ServerSetup.md.vm * (edit) hadoop-project/pom.xml > Upgrade Tomcat to 6.0.48 > > > Key: HADOOP-13812 > URL: https://issues.apache.org/jira/browse/HADOOP-13812 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Blocker > Attachments: HADOOP-13812.001.patch > > > KMS and HttpFS currently uses Tomcat 6.0.44, propose to upgrade to the latest > version is 6.0.48. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
