[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501321#comment-16501321 ] Akira Ajisaka commented on HADOOP-15482: +1 > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15514) NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
[
https://issues.apache.org/jira/browse/HADOOP-15514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S reassigned HADOOP-15514:
--
Assignee: Rohith Sharma K S
> NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
> --
>
> Key: HADOOP-15514
> URL: https://issues.apache.org/jira/browse/HADOOP-15514
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Rohith Sharma K S
>Priority: Major
>
> {code:java}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15514) NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
[
https://issues.apache.org/jira/browse/HADOOP-15514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15514:
Issue Type: Sub-task (was: Bug)
Parent: HADOOP-15139
> NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
> --
>
> Key: HADOOP-15514
> URL: https://issues.apache.org/jira/browse/HADOOP-15514
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Priority: Major
>
> {code:java}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15514) NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
[
https://issues.apache.org/jira/browse/HADOOP-15514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15514:
Affects Version/s: 3.0.0
> NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
> --
>
> Key: HADOOP-15514
> URL: https://issues.apache.org/jira/browse/HADOOP-15514
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Priority: Major
>
> {code:java}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15514) NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
[
https://issues.apache.org/jira/browse/HADOOP-15514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501286#comment-16501286
]
Rohith Sharma K S commented on HADOOP-15514:
It looks shaded client jar doesn't include timeline-service jar.
> NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
> --
>
> Key: HADOOP-15514
> URL: https://issues.apache.org/jira/browse/HADOOP-15514
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Jeff Zhang
>Priority: Major
>
> {code:java}
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975){code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501278#comment-16501278 ] Hudson commented on HADOOP-15507: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14362 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14362/]) HADOOP-15507. Add MapReduce counters about EC bytes read. (xiao: rev 6d5e87aec2f615ed265dc495873bf53ee7d2ace2) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/FileSystemCounter.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystemStorageStatistics.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/org/apache/hadoop/mapreduce/FileSystemCounter.properties * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemStorageStatistics.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReaderStrategy.java > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497253#comment-16497253 ] Xiao Chen edited comment on HADOOP-15507 at 6/5/18 4:14 AM: Updating patch 1 to demonstrate the idea and solicit feedback was (Author: xiaochen): Updating patch 1 to demonstrate the idea and solicit early feedback > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15507: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk, thanks again for the reviews! > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.2.0 > > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501266#comment-16501266 ] Xiao Chen commented on HADOOP-15507: Thanks for the review Haibo and Fabbri. bq. manually tested (i.e. your screenshot is real) Yes, tested this in a real cluster and the picture is a pure screenshot of a job's counter - no image editing. :) Based on the +1's, I'm committing this. Failed TestTrash looks unrelated and passed locally. > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[
https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501264#comment-16501264
]
genericqa commented on HADOOP-15513:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-common-project/hadoop-common: The patch
generated 3 new + 59 unchanged - 0 fixed = 62 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15513 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12926490/HADOOP-15513.v1.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 55de65de1b93 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8d31ddc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14723/artifact/out/diff-checkstyle-hadoop-common-project_hadoop-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14723/testReport/ |
| Max. process+thread count | 1764 (vs. ulimit of 1) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14723/console |
| Powered by | Apache Yetus
[jira] [Created] (HADOOP-15514) NoClassDefFoundError of TimelineCollectorManager when starting MiniCluster
Jeff Zhang created HADOOP-15514:
---
Summary: NoClassDefFoundError of TimelineCollectorManager when
starting MiniCluster
Key: HADOOP-15514
URL: https://issues.apache.org/jira/browse/HADOOP-15514
Project: Hadoop Common
Issue Type: Bug
Reporter: Jeff Zhang
{code:java}
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/server/timelineservice/collector/TimelineCollectorManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethods(Class.java:1975){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun reassigned HADOOP-10768:
---
Assignee: Wei-Chiu Chuang (was: Dapeng Sun)
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
> Optimize Hadoop RPC encryption performance.pdf,
> cpu_profile_RPC_encryption_AES.png,
> cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501196#comment-16501196 ] Íñigo Goiri commented on HADOOP-15513: -- Regarding [^HADOOP-15513.v1.patch], I have one high level comment. Not all the things tested here are errors right now (e.g., case 3). We may want to split testSymlinkErrorCases into 5 test cases (e.g., testSymLinkAlreadyExists()). Then we can add the full description of each test in each javadoc function. We should also put these tests right after the symlink tests and not in the end. > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > > Add additional test cases to cover some corner cases for FileUtil#symlink. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501165#comment-16501165 ] Íñigo Goiri commented on HADOOP-15513: -- This should be useful to verify that patches like HADOOP-15465 don't break compatibility. > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > > Add additional test cases to cover some corner cases for FileUtil#symlink. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HADOOP-15513: - Description: Add additional test cases to cover some corner cases for FileUtil#symlink. > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > > Add additional test cases to cover some corner cases for FileUtil#symlink. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15465) Deprecate WinUtils#Symlinks by using native java code
[ https://issues.apache.org/jira/browse/HADOOP-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501154#comment-16501154 ] Giovanni Matteo Fumarola commented on HADOOP-15465: --- Thanks [~ste...@apache.org] and [~elgoiri] for the feedback. I wrote some additional unit tests for FileUtil#symlink in HADOOP-15513. > Deprecate WinUtils#Symlinks by using native java code > - > > Key: HADOOP-15465 > URL: https://issues.apache.org/jira/browse/HADOOP-15465 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15465.v0.patch, HADOOP-15465.v0.proto.patch, > HADOOP-15465.v1.patch, HADOOP-15465.v2.patch, HADOOP-15465.v3.patch > > > Hadoop uses the shell to create symbolic links. Now that Hadoop relies on > Java 7+, we can deprecate all the shell code and rely on the Java APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15513: -- Status: Patch Available (was: Open) > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15513: -- Attachment: HADOOP-15513.v1.patch > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15513) Add additional test cases to cover some corner cases for FileUtil#symlink
[ https://issues.apache.org/jira/browse/HADOOP-15513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated HADOOP-15513: -- Summary: Add additional test cases to cover some corner cases for FileUtil#symlink (was: Add additional test cases to over corner cases for FileUtil#symlink) > Add additional test cases to cover some corner cases for FileUtil#symlink > - > > Key: HADOOP-15513 > URL: https://issues.apache.org/jira/browse/HADOOP-15513 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15513.v1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15513) Add additional test cases to over corner cases for FileUtil#symlink
Giovanni Matteo Fumarola created HADOOP-15513: - Summary: Add additional test cases to over corner cases for FileUtil#symlink Key: HADOOP-15513 URL: https://issues.apache.org/jira/browse/HADOOP-15513 Project: Hadoop Common Issue Type: Sub-task Reporter: Giovanni Matteo Fumarola Assignee: Giovanni Matteo Fumarola -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15465) Deprecate WinUtils#Symlinks by using native java code
[ https://issues.apache.org/jira/browse/HADOOP-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501064#comment-16501064 ] Íñigo Goiri commented on HADOOP-15465: -- [~giovanni.fumarola], can you add unit tests to check for the failure cases? In that way we can make sure we are not breaking compatibility. Once we have the approach working, we can change the semantic and the unit tests. > Deprecate WinUtils#Symlinks by using native java code > - > > Key: HADOOP-15465 > URL: https://issues.apache.org/jira/browse/HADOOP-15465 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15465.v0.patch, HADOOP-15465.v0.proto.patch, > HADOOP-15465.v1.patch, HADOOP-15465.v2.patch, HADOOP-15465.v3.patch > > > Hadoop uses the shell to create symbolic links. Now that Hadoop relies on > Java 7+, we can deprecate all the shell code and rely on the Java APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15137:
Fix Version/s: 3.1.1
3.2.0
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15137:
Target Version/s: (was: 3.2.0, 3.1.1)
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501056#comment-16501056
]
Bharat Viswanadham commented on HADOOP-15137:
-
Committed to branch-3, branch-3.1, and trunk.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HADOOP-15137:
Resolution: Fixed
Target Version/s: 3.2.0, 3.1.1
Status: Resolved (was: Patch Available)
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501055#comment-16501055
]
Hudson commented on HADOOP-15137:
-
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14359 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/14359/])
HADOOP-15137. ClassNotFoundException: (bharat: rev
f30f2dc4085c51b6b6850430213f47b97a763b7f)
* (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[
https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501034#comment-16501034
]
Xiao Chen commented on HADOOP-15217:
Thanks very much for the thoughts here Zsolt, and the discussion offline.
To summarize, I think the core problem is, {{URL#openStream}} can throw IOE for
all kinds of reasons, so we're not explicitly wrapping the fix in this patch.
The added test is valid to make sure there is no regression, and in case of a
regression, I'd argue changing the exception type from IOE to AssertError
doesn't change the probability of a developer ignoring the test. For the same
reason, we don't wrap filesystem.create calls to make sure it doesn't regress.
:)
So given the extra 6 lines of code doesn't make debugging easier, we should aim
for readability and let the underlying exception fail the test in case of a
regression.
> org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
> --
>
> Key: HADOOP-15217
> URL: https://issues.apache.org/jira/browse/HADOOP-15217
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 3.0.0
>Reporter: Joseph Fourny
>Assignee: Zsolt Venczel
>Priority: Major
> Attachments: HADOOP-15217.01.patch, HADOOP-15217.02.patch,
> HADOOP-15217.03.patch, HADOOP-15217.04.patch, HADOOP-15217.05.patch,
> TestCase.java
>
>
> When _FsUrlStreamHandlerFactory_ is registered with _java.net.URL_ (ex: when
> Spark is initialized), it breaks URLs with spaces (even though they are
> properly URI-encoded). I traced the problem down to
> _FSUrlConnection.connect()_ method. It naively gets the path from the URL,
> which contains encoded spaces, and pases it to
> _org.apache.hadoop.fs.Path(String)_ constructor. This is not correct, because
> the docs clearly say that the string must NOT be encoded. Doing so causes
> double encoding within the Path class (ie: %20 becomes %2520).
> See attached JUnit test.
> This test case mimics an issue I ran into when trying to use Commons
> Configuration 1.9 AFTER initializing Spark. Commons Configuration uses URL
> class to load configuration files, but Spark installs
> _FsUrlStreamHandlerFactory_, which hits this issue. For now, we are using an
> AspectJ aspect to "patch" the bytecode at load time to work-around the issue.
> The real fix is quite simple. All you need to do is replace this line in
> _org.apache.hadoop.fs.FsUrlConnection.connect()_:
> is = fs.open(new Path(url.getPath()));
> with this line:
> is = fs.open(new Path(url.*toUri()*.getPath()));
> URI.getPath() will correctly decode the path, which is what is expected by
> _org.apache.hadoop.fs.Path(String)_ constructor.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500978#comment-16500978
]
Bharat Viswanadham edited comment on HADOOP-15137 at 6/4/18 10:43 PM:
--
Thank You [~jnp] for review.
I will commit this patch shortly.
was (Author: bharatviswa):
I will commit this patch shortly.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500978#comment-16500978
]
Bharat Viswanadham commented on HADOOP-15137:
-
I will commit this patch shortly.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500930#comment-16500930
]
genericqa commented on HADOOP-10768:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 33m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 33m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 33m
12s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 19s{color} | {color:orange} root: The patch generated 28 new + 904 unchanged
- 14 fixed = 932 total (was 918) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
13s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
39s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 96m 41s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}255m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-10768 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12926426/HADOOP-10768.011.patch
|
|
[jira] [Commented] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500916#comment-16500916 ] Aaron Fabbri commented on HADOOP-15507: --- +1 assuming you manually tested (i.e. your screenshot is real) in addition to running the updated unit test in the patch. Patch looks fine to me. > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15484) Upgrade moment.js to version 2.22.1
[ https://issues.apache.org/jira/browse/HADOOP-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500790#comment-16500790 ] Jitendra Nath Pandey commented on HADOOP-15484: --- The patch looks good to me. +1 > Upgrade moment.js to version 2.22.1 > --- > > Key: HADOOP-15484 > URL: https://issues.apache.org/jira/browse/HADOOP-15484 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15484.001.patch > > > This Jira aims to upgrade moment.js to version 2.22.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15483) Upgrade jquery to version 3.3.1
[ https://issues.apache.org/jira/browse/HADOOP-15483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500791#comment-16500791 ] Jitendra Nath Pandey commented on HADOOP-15483: --- +1 for the patch. > Upgrade jquery to version 3.3.1 > --- > > Key: HADOOP-15483 > URL: https://issues.apache.org/jira/browse/HADOOP-15483 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15483.001.patch, HADOOP-15483.002.patch, > HADOOP-15483.003.patch, HADOOP-15483.004.patch, HADOOP-15483.005.patch, > HADOOP-15483.006.patch > > > This Jira aims to upgrade jquery to version 3.3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-12649) Improve Kerberos diagnostics and failure handling
[ https://issues.apache.org/jira/browse/HADOOP-12649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500713#comment-16500713 ] Marouane BENALLA edited comment on HADOOP-12649 at 6/4/18 7:05 PM: --- Hello [~ste...@apache.org], Should I do something before working on a ticket? I'd like to work on HADOOP-12741 was (Author: m.benalla): Hello Steve Loughran, Should I do something before working on a ticket? I'd like to work on HADOOP-12741 > Improve Kerberos diagnostics and failure handling > - > > Key: HADOOP-12649 > URL: https://issues.apache.org/jira/browse/HADOOP-12649 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.1 > Environment: Kerberos >Reporter: Steve Loughran >Priority: Major > > Sometimes —apparently— some people cannot get kerberos to work. > The ability to diagnose problems here is hampered by some aspects of UGI > # the only way to turn on JAAS debug information is through an env var, not > within the JVM > # failures are potentially underlogged > # exceptions raised are generic IOEs, so can't be trapped and filtered > # failure handling on the TGT renewer thread is nonexistent > # the code is barely-readable, underdocumented mess. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12649) Improve Kerberos diagnostics and failure handling
[ https://issues.apache.org/jira/browse/HADOOP-12649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500713#comment-16500713 ] Marouane BENALLA commented on HADOOP-12649: --- Hello Steve Loughran, Should I do something before working on a ticket? I'd like to work on HADOOP-12741 > Improve Kerberos diagnostics and failure handling > - > > Key: HADOOP-12649 > URL: https://issues.apache.org/jira/browse/HADOOP-12649 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Affects Versions: 2.7.1 > Environment: Kerberos >Reporter: Steve Loughran >Priority: Major > > Sometimes —apparently— some people cannot get kerberos to work. > The ability to diagnose problems here is hampered by some aspects of UGI > # the only way to turn on JAAS debug information is through an env var, not > within the JVM > # failures are potentially underlogged > # exceptions raised are generic IOEs, so can't be trapped and filtered > # failure handling on the TGT renewer thread is nonexistent > # the code is barely-readable, underdocumented mess. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500669#comment-16500669
]
Daryn Sharp commented on HADOOP-10768:
--
{quote}Yes, shouldAuthenticateOverKrb() would throw NPE in some cases, we
should catch it.
{quote}
No, catching and ignoring all exceptions from shouldAuthenticateOverKrb is
completely wrong.
I appreciate all the work here. However I have a number of security concerns.
I didn't care too much before other than not breaking the protocol and not
making simple mistakes. Times change. I care a lot now.
I don't fully understand what's going on in the key negotiation, don't feel
qualified to determine if it's secure (although it doesn't appear to be), doubt
few in our community are either, now and in the future. I don't like being
tied to AES/CTR+custom-hmac when other ciphers like AES/GCM/ECDHE may be a
requirement.
An actual SSL implementation written by experts that we don't maintain and is
tuned for performance is a far preferable approach. To that end, I've made
surprising small tweaks to the ipc server to use netty (configurable). Goal is
dropping in netty's SSL handlers.
Note that hadoop's ipc is pretty darn optimized. I was surprised to find netty
initially incurred a 20% performance hit. I've since closed the gap to <5%.
Note that one of the challenges was avoiding array copies and allocations. I
see so much in the current patch that it's likely what's sapping performance so
much.
I'm finishing up debugging a few failed test cases. Should post on a different
Jira this afternoon. Then we can work on the ipc client.
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
>Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
> Optimize Hadoop RPC encryption performance.pdf,
> cpu_profile_RPC_encryption_AES.png,
> cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15507: Attachment: (was: YARN-8240-YARN-1011.02.patch) > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated HADOOP-15507: Attachment: YARN-8240-YARN-1011.02.patch > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, YARN-8240-YARN-1011.02.patch, > image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15507) Add MapReduce counters about EC bytes read
[ https://issues.apache.org/jira/browse/HADOOP-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500612#comment-16500612 ] Haibo Chen commented on HADOOP-15507: - The MapReduce part looks good to me, but I'm not very familiar with the erase code or the file system API to comment on the rest of the patch. > Add MapReduce counters about EC bytes read > -- > > Key: HADOOP-15507 > URL: https://issues.apache.org/jira/browse/HADOOP-15507 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15507.01.patch, image-2018-05-31-15-29-45-729.png > > > HDFS has added Erasure Coding support in HDFS-7285. There are HDFS level > [ReadStatistics|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReadStatistics.java] > so from DFSClient we can know how much reads are EC/replication. > In order for users to have a better view of how much of their workload is > impacted by EC, we can expose EC read bytes to File System Counters, and to > MapReduce's job counters. This way, end users can tell from MR jobs directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500601#comment-16500601
]
Wei-Chiu Chuang commented on HADOOP-10768:
--
Ooops didn't realize there's a code conflict due to HDFS-13601. Upload a new
rev (v011 [^HADOOP-10768.011.patch] ) to address the conflict.
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
>Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
> Optimize Hadoop RPC encryption performance.pdf,
> cpu_profile_RPC_encryption_AES.png,
> cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15137) ClassNotFoundException: org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using hadoop-client-minicluster
[
https://issues.apache.org/jira/browse/HADOOP-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500603#comment-16500603
]
Jitendra Nath Pandey commented on HADOOP-15137:
---
I think this patch is ready to be committed +1.
Any further enhancements needed should be added as subtasks for the parent jira.
> ClassNotFoundException:
> org.apache.hadoop.yarn.server.api.DistributedSchedulingAMProtocol when using
> hadoop-client-minicluster
> --
>
> Key: HADOOP-15137
> URL: https://issues.apache.org/jira/browse/HADOOP-15137
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Jeff Zhang
>Assignee: Bharat Viswanadham
>Priority: Major
> Attachments: HADOOP-15137.01.patch, HADOOP-15137.02.patch,
> YARN-7673.00.patch
>
>
> I'd like to use hadoop-client-minicluster for hadoop downstream project, but
> I encounter the following exception when starting hadoop minicluster. And I
> check the hadoop-client-minicluster, it indeed does not have this class. Is
> this something that is missing when packaging the published jar ?
> {code}
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/server/api/DistributedSchedulingAMProtocol
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.createResourceManager(MiniYARNCluster.java:851)
> at
> org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:285)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-10768) Optimize Hadoop RPC encryption performance
[
https://issues.apache.org/jira/browse/HADOOP-10768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HADOOP-10768:
-
Attachment: HADOOP-10768.011.patch
> Optimize Hadoop RPC encryption performance
> --
>
> Key: HADOOP-10768
> URL: https://issues.apache.org/jira/browse/HADOOP-10768
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, security
>Affects Versions: 3.0.0-alpha1
>Reporter: Yi Liu
>Assignee: Dapeng Sun
>Priority: Major
> Attachments: HADOOP-10768.001.patch, HADOOP-10768.002.patch,
> HADOOP-10768.003.patch, HADOOP-10768.004.patch, HADOOP-10768.005.patch,
> HADOOP-10768.006.patch, HADOOP-10768.007.patch, HADOOP-10768.008.patch,
> HADOOP-10768.009.patch, HADOOP-10768.010.patch, HADOOP-10768.011.patch,
> Optimize Hadoop RPC encryption performance.pdf,
> cpu_profile_RPC_encryption_AES.png,
> cpu_profile_rpc_encryption_optimize_calculateHMAC.png
>
>
> Hadoop RPC encryption is enabled by setting {{hadoop.rpc.protection}} to
> "privacy". It utilized SASL {{GSSAPI}} and {{DIGEST-MD5}} mechanisms for
> secure authentication and data protection. Even {{GSSAPI}} supports using
> AES, but without AES-NI support by default, so the encryption is slow and
> will become bottleneck.
> After discuss with [~atm], [~tucu00] and [~umamaheswararao], we can do the
> same optimization as in HDFS-6606. Use AES-NI with more than *20x* speedup.
> On the other hand, RPC message is small, but RPC is frequent and there may be
> lots of RPC calls in one connection, we needs to setup benchmark to see real
> improvement and then make a trade-off.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500550#comment-16500550 ] Steve Loughran commented on HADOOP-15482: - OK. If you've done the tests, I'm +1 on this > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-12707) key of FileSystem inner class Cache contains UGI.hascode which uses the default hascode method, leading to the memory leak with Proxy Users
[
https://issues.apache.org/jira/browse/HADOOP-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500549#comment-16500549
]
Steve Loughran commented on HADOOP-12707:
-
Added proxy users to the title.
[~cnauroth]: what are your thoughts here?
[~xinlishang] I think you may want to consider what you could do with your own
cache, as that would isolate your code & you could be clever here. That is; you
can cache filesystems outside FileSystem.cache if you really want to
> key of FileSystem inner class Cache contains UGI.hascode which uses the
> default hascode method, leading to the memory leak with Proxy Users
> ---
>
> Key: HADOOP-12707
> URL: https://issues.apache.org/jira/browse/HADOOP-12707
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1
>Reporter: sunhaitao
>Assignee: sunhaitao
>Priority: Major
>
> FileSystem.get(conf) method,By default it will get the fs object from
> CACHE,But the key of the CACHE constains ugi.hashCode, which uses the
> default hascode method of subject instead of the hascode method overwritten
> by subject.
>@Override
> public int hashCode() {
> return (scheme + authority).hashCode() + ugi.hashCode() + (int)unique;
> }
> In this case, even if same user, if the calll FileSystem.get(conf) twice, two
> different key will be created. In long duartion, this will lead to memory
> leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-12707) key of FileSystem inner class Cache contains UGI.hascode which uses the default hascode method, leading to the memory leak with Proxy Users
[
https://issues.apache.org/jira/browse/HADOOP-12707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-12707:
Summary: key of FileSystem inner class Cache contains UGI.hascode which
uses the default hascode method, leading to the memory leak with Proxy Users
(was: key of FileSystem inner class Cache contains UGI.hascode which uses the
defualt hascode method, leading to the memory leak)
> key of FileSystem inner class Cache contains UGI.hascode which uses the
> default hascode method, leading to the memory leak with Proxy Users
> ---
>
> Key: HADOOP-12707
> URL: https://issues.apache.org/jira/browse/HADOOP-12707
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.7.1
>Reporter: sunhaitao
>Assignee: sunhaitao
>Priority: Major
>
> FileSystem.get(conf) method,By default it will get the fs object from
> CACHE,But the key of the CACHE constains ugi.hashCode, which uses the
> default hascode method of subject instead of the hascode method overwritten
> by subject.
>@Override
> public int hashCode() {
> return (scheme + authority).hashCode() + ugi.hashCode() + (int)unique;
> }
> In this case, even if same user, if the calll FileSystem.get(conf) twice, two
> different key will be created. In long duartion, this will lead to memory
> leak.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15465) Deprecate WinUtils#Symlinks by using native java code
[ https://issues.apache.org/jira/browse/HADOOP-15465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500544#comment-16500544 ] Steve Loughran commented on HADOOP-15465: - [~elgoiri] if that's what you want, I'll go with that. I'll let you do the review & test... > Deprecate WinUtils#Symlinks by using native java code > - > > Key: HADOOP-15465 > URL: https://issues.apache.org/jira/browse/HADOOP-15465 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Giovanni Matteo Fumarola >Priority: Major > Attachments: HADOOP-15465.v0.patch, HADOOP-15465.v0.proto.patch, > HADOOP-15465.v1.patch, HADOOP-15465.v2.patch, HADOOP-15465.v3.patch > > > Hadoop uses the shell to create symbolic links. Now that Hadoop relies on > Java 7+, we can deprecate all the shell code and rely on the Java APIs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15482) Upgrade jackson-databind to version 2.9.5
[ https://issues.apache.org/jira/browse/HADOOP-15482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500533#comment-16500533 ] Jitendra Nath Pandey commented on HADOOP-15482: --- I am +1 to commit this, unless there are objections, will wait for a couple of days. > Upgrade jackson-databind to version 2.9.5 > - > > Key: HADOOP-15482 > URL: https://issues.apache.org/jira/browse/HADOOP-15482 > Project: Hadoop Common > Issue Type: Task >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HADOOP-15482.001.patch, HADOOP-15482.002.patch > > > This Jira aims to upgrade jackson-databind to version 2.9.5 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15493) DiskChecker should handle disk full situation
[ https://issues.apache.org/jira/browse/HADOOP-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500356#comment-16500356 ] Daryn Sharp commented on HADOOP-15493: -- Exception message strings should be considered as opaque. Checking for a substring in the message is incredibly brittle as evident by the linux/window checks. I'd just remove it. > DiskChecker should handle disk full situation > - > > Key: HADOOP-15493 > URL: https://issues.apache.org/jira/browse/HADOOP-15493 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Arpit Agarwal >Assignee: Arpit Agarwal >Priority: Critical > Attachments: HADOOP-15493.01.patch, HADOOP-15493.02.patch > > > DiskChecker#checkDirWithDiskIo creates a file to verify that the disk is > writable. > However check should not fail when file creation fails due to disk being > full. This avoids marking full disks as _failed_. > Reported by [~kihwal] and [~daryn] in HADOOP-15450. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15512) clean up Shell from JDK7 workarounds
Steve Loughran created HADOOP-15512:
---
Summary: clean up Shell from JDK7 workarounds
Key: HADOOP-15512
URL: https://issues.apache.org/jira/browse/HADOOP-15512
Project: Hadoop Common
Issue Type: Improvement
Components: common
Affects Versions: 3.1.0
Reporter: Steve Loughran
there's some comments in {{Shell}} about JDK7 specific issues (especially
{{runCommand()}}. These workarounds don't matter, so can be purged
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15487) ConcurrentModificationException resulting in Kerberos authentication error.
[
https://issues.apache.org/jira/browse/HADOOP-15487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500348#comment-16500348
]
Daryn Sharp commented on HADOOP-15487:
--
The second exception is an unrelated jdk bug fixed in 8u161. [JDK-8170278:
ticket renewal won't happen with debugging turned
on|https://bugs.openjdk.java.net/browse/JDK-8170278]. The gssapi is smart
recognizes and handles expired tickets from a keytab. The problem is
{{KerberosTicket#toString}} throws the ISE if it's expired. Easy workaround is
don't enable debug logging.
The original issue is distinct. If there truly are no custom plugins, it may
be related to curator/zookeeper/AuthenticatedURL. What is the specific apache
release? Did the server recover?
We may need to consider using a distinct subject/ugi for rpc servers to prevent
other code munging our JASS, but there are a few possible grues lurking there.
> ConcurrentModificationException resulting in Kerberos authentication error.
> ---
>
> Key: HADOOP-15487
> URL: https://issues.apache.org/jira/browse/HADOOP-15487
> Project: Hadoop Common
> Issue Type: Bug
> Environment: CDH 5.13.3. Kerberized, Hadoop-HA, jdk1.8.0_152
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> We found the following exception message in a NameNode log. It seems the
> ConcurrentModificationException caused Kerberos authentication error.
> It appears to be a JDK bug, similar to HADOOP-13433 (Race in
> UGI.reloginFromKeytab) but the version of Hadoop (CDH5.13.3) already patched
> HADOOP-13433. (The stacktrace also differs) This cluster runs on JDK
> 1.8.0_152.
> {noformat}
> 2018-05-19 04:00:00,182 WARN org.apache.hadoop.security.UserGroupInformation:
> PriviledgedActionException as:hdfs/no...@example.com (auth:KERBEROS)
> cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> 2018-05-19 04:00:00,183 INFO org.apache.hadoop.ipc.Server: Socket Reader #1
> for port 8020: readAndProcess from client 10.16.20.122 threw exception
> [java.util.ConcurrentModificationException]
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at javax.security.auth.Subject$SecureSet$1.next(Subject.java:1070)
> at javax.security.auth.Subject$ClassSet$1.run(Subject.java:1401)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1399)
> at javax.security.auth.Subject$ClassSet.(Subject.java:1372)
> at javax.security.auth.Subject.getPrivateCredentials(Subject.java:767)
> at
> sun.security.jgss.krb5.SubjectComber.findAux(SubjectComber.java:127)
> at
> sun.security.jgss.krb5.SubjectComber.findMany(SubjectComber.java:69)
> at
> sun.security.jgss.krb5.ServiceCreds.getInstance(ServiceCreds.java:96)
> at sun.security.jgss.krb5.Krb5Util.getServiceCreds(Krb5Util.java:203)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:74)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential$1.run(Krb5AcceptCredential.java:72)
> at java.security.AccessController.doPrivileged(Native Method)
> at
> sun.security.jgss.krb5.Krb5AcceptCredential.getInstance(Krb5AcceptCredential.java:71)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:127)
> at
> sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:193)
> at sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:427)
> at
> sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:62)
> at
> sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:154)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108)
> at
> com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at
> org.apache.hadoop.security.SaslRpcServer$FastSaslServerFactory.createSaslServer(SaslRpcServer.java:398)
> at
> org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:164)
> at
> org.apache.hadoop.security.SaslRpcServer$1.run(SaslRpcServer.java:161)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> at
> org.apache.hadoop.security.SaslRpcServer.create(SaslRpcServer.java:160)
> at
>
[jira] [Resolved] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence
[
https://issues.apache.org/jira/browse/HADOOP-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota resolved HADOOP-15511.
-
Resolution: Not A Problem
Sorry for the noise on this, as it turns out the test is working as expected.
The following happens:
* Tests a circular sequence of renames to verify that overwriting recently
* deleted files and reading recently created files from rename operations
* works as expected.
The first check of the directory (the first doTestRenameSequence) won't check
anything, but the followups are working as expected.
> ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
> doTestRenameSequence
>
>
> Key: HADOOP-15511
> URL: https://issues.apache.org/jira/browse/HADOOP-15511
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Bumped into this while working on HADOOP-15423.
> Currently the parameters passed to doTestRenameSequence are the following:
> {noformat}
> mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> {noformat}
> The problem is that we should check that the srcdir is moved to dstdirs, so
> it should be check in nodirs. Right now nodirs parameter checks for
> directories which are completely unrelated to the cases, so this part of the
> test should be fixed to check for dirs that are related to the cases.
> Basically the nodirs should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[
https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500292#comment-16500292
]
genericqa commented on HADOOP-15217:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 28m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
26s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
36s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12926359/HADOOP-15217.05.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 38d336311845 3.13.0-137-generic #186-Ubuntu SMP Mon Dec 4
19:09:19 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9efb4b7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HADOOP-Build/14721/testReport/ |
| Max. process+thread count | 1468 (vs. ulimit of 1) |
| modules |
[jira] [Updated] (HADOOP-15307) Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
[
https://issues.apache.org/jira/browse/HADOOP-15307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15307:
Status: In Progress (was: Patch Available)
> Improve NFS error handling: Unsupported verifier flavorAUTH_SYS
> ---
>
> Key: HADOOP-15307
> URL: https://issues.apache.org/jira/browse/HADOOP-15307
> Project: Hadoop Common
> Issue Type: Bug
> Components: nfs
> Environment: CentOS 7.4, CDH5.13.1, Kerberized Hadoop cluster
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Major
> Labels: newbie
> Attachments: HADOOP-15307.001.patch, HADOOP-15307.002.patch
>
>
> When NFS gateway starts and if the portmapper request is denied by rpcbind
> for any reason (in our case, /etc/hosts.allow did not have the localhost),
> NFS gateway fails with the following obscure exception:
> {noformat}
> 2018-03-05 12:49:31,976 INFO org.apache.hadoop.oncrpc.SimpleUdpServer:
> Started listening to UDP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,988 INFO org.apache.hadoop.oncrpc.SimpleTcpServer:
> Started listening to TCP requests at port 4242 for Rpc program: mountd at
> localhost:4242 with workerCount 1
> 2018-03-05 12:49:31,993 TRACE org.apache.hadoop.oncrpc.RpcCall:
> Xid:692394656, messageType:RPC_CALL, rpcVersion:2, program:10, version:2,
> procedure:1, credential:(AuthFlavor:AUTH_NONE),
> verifier:(AuthFlavor:AUTH_NONE)
> 2018-03-05 12:49:31,998 FATAL org.apache.hadoop.mount.MountdBase: Failed to
> start the server. Cause:
> java.lang.UnsupportedOperationException: Unsupported verifier flavorAUTH_SYS
> at
> org.apache.hadoop.oncrpc.security.Verifier.readFlavorAndVerifier(Verifier.java:45)
> at org.apache.hadoop.oncrpc.RpcDeniedReply.read(RpcDeniedReply.java:50)
> at org.apache.hadoop.oncrpc.RpcReply.read(RpcReply.java:67)
> at org.apache.hadoop.oncrpc.SimpleUdpClient.run(SimpleUdpClient.java:71)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:130)
> at org.apache.hadoop.oncrpc.RpcProgram.register(RpcProgram.java:101)
> at org.apache.hadoop.mount.MountdBase.start(MountdBase.java:83)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startServiceInternal(Nfs3.java:56)
> at org.apache.hadoop.hdfs.nfs.nfs3.Nfs3.startService(Nfs3.java:69)
> at
> org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter.start(PrivilegedNfsGatewayStarter.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
> 2018-03-05 12:49:32,007 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1{noformat}
> Reading the code comment for class Verifier, I think this bug existed since
> its inception
> {code:java}
> /**
> * Base class for verifier. Currently our authentication only supports 3 types
> * of auth flavors: {@link RpcAuthInfo.AuthFlavor#AUTH_NONE}, {@link
> RpcAuthInfo.AuthFlavor#AUTH_SYS},
> * and {@link RpcAuthInfo.AuthFlavor#RPCSEC_GSS}. Thus for verifier we only
> need to handle
> * AUTH_NONE and RPCSEC_GSS
> */
> public abstract class Verifier extends RpcAuthInfo {{code}
> The verifier should also handle AUTH_SYS too.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[ https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel updated HADOOP-15217: --- Attachment: HADOOP-15217.05.patch > org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces > -- > > Key: HADOOP-15217 > URL: https://issues.apache.org/jira/browse/HADOOP-15217 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Joseph Fourny >Assignee: Zsolt Venczel >Priority: Major > Attachments: HADOOP-15217.01.patch, HADOOP-15217.02.patch, > HADOOP-15217.03.patch, HADOOP-15217.04.patch, HADOOP-15217.05.patch, > TestCase.java > > > When _FsUrlStreamHandlerFactory_ is registered with _java.net.URL_ (ex: when > Spark is initialized), it breaks URLs with spaces (even though they are > properly URI-encoded). I traced the problem down to > _FSUrlConnection.connect()_ method. It naively gets the path from the URL, > which contains encoded spaces, and pases it to > _org.apache.hadoop.fs.Path(String)_ constructor. This is not correct, because > the docs clearly say that the string must NOT be encoded. Doing so causes > double encoding within the Path class (ie: %20 becomes %2520). > See attached JUnit test. > This test case mimics an issue I ran into when trying to use Commons > Configuration 1.9 AFTER initializing Spark. Commons Configuration uses URL > class to load configuration files, but Spark installs > _FsUrlStreamHandlerFactory_, which hits this issue. For now, we are using an > AspectJ aspect to "patch" the bytecode at load time to work-around the issue. > The real fix is quite simple. All you need to do is replace this line in > _org.apache.hadoop.fs.FsUrlConnection.connect()_: > is = fs.open(new Path(url.getPath())); > with this line: > is = fs.open(new Path(url.*toUri()*.getPath())); > URI.getPath() will correctly decode the path, which is what is expected by > _org.apache.hadoop.fs.Path(String)_ constructor. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence
[
https://issues.apache.org/jira/browse/HADOOP-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15511:
Affects Version/s: (was: 3.2.0)
3.1.0
> ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
> doTestRenameSequence
>
>
> Key: HADOOP-15511
> URL: https://issues.apache.org/jira/browse/HADOOP-15511
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.1.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Bumped into this while working on HADOOP-15423.
> Currently the parameters passed to doTestRenameSequence are the following:
> {noformat}
> mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> {noformat}
> The problem is that we should check that the srcdir is moved to dstdirs, so
> it should be check in nodirs. Right now nodirs parameter checks for
> directories which are completely unrelated to the cases, so this part of the
> test should be fixed to check for dirs that are related to the cases.
> Basically the nodirs should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence
[
https://issues.apache.org/jira/browse/HADOOP-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15511:
Affects Version/s: 3.2.0
> ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
> doTestRenameSequence
>
>
> Key: HADOOP-15511
> URL: https://issues.apache.org/jira/browse/HADOOP-15511
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.2.0
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Bumped into this while working on HADOOP-15423.
> Currently the parameters passed to doTestRenameSequence are the following:
> {noformat}
> mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> {noformat}
> The problem is that we should check that the srcdir is moved to dstdirs, so
> it should be check in nodirs. Right now nodirs parameter checks for
> directories which are completely unrelated to the cases, so this part of the
> test should be fixed to check for dirs that are related to the cases.
> Basically the nodirs should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence
[
https://issues.apache.org/jira/browse/HADOOP-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gabor Bota updated HADOOP-15511:
Summary: ITestS3GuardListConsistency#testRollingRenames bad parameters
passed to doTestRenameSequence (was:
ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
doTestRenameSequence doTestRenameSequence)
> ITestS3GuardListConsistency#testRollingRenames bad parameters passed to
> doTestRenameSequence
>
>
> Key: HADOOP-15511
> URL: https://issues.apache.org/jira/browse/HADOOP-15511
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Gabor Bota
>Assignee: Gabor Bota
>Priority: Major
>
> Bumped into this while working on HADOOP-15423.
> Currently the parameters passed to doTestRenameSequence are the following:
> {noformat}
> mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> mkdirs: {}
> srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
> dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
> nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
> {noformat}
> The problem is that we should check that the srcdir is moved to dstdirs, so
> it should be check in nodirs. Right now nodirs parameter checks for
> directories which are completely unrelated to the cases, so this part of the
> test should be fixed to check for dirs that are related to the cases.
> Basically the nodirs should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15511) ITestS3GuardListConsistency#testRollingRenames bad parameters passed to doTestRenameSequence doTestRenameSequence
Gabor Bota created HADOOP-15511:
---
Summary: ITestS3GuardListConsistency#testRollingRenames bad
parameters passed to doTestRenameSequence doTestRenameSequence
Key: HADOOP-15511
URL: https://issues.apache.org/jira/browse/HADOOP-15511
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Gabor Bota
Assignee: Gabor Bota
Bumped into this while working on HADOOP-15423.
Currently the parameters passed to doTestRenameSequence are the following:
{noformat}
mkdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
mkdirs: {}
srcdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/1
dstdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
yesdirs: s3a://cloudera-dev-gabor-ireland/test/rolling/2
nodirs: s3a://cloudera-dev-gabor-ireland/test/rolling/3
{noformat}
The problem is that we should check that the srcdir is moved to dstdirs, so it
should be check in nodirs. Right now nodirs parameter checks for directories
which are completely unrelated to the cases, so this part of the test should be
fixed to check for dirs that are related to the cases. Basically the nodirs
should be equal to srcdirs in this case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[
https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500103#comment-16500103
]
genericqa commented on HADOOP-15217:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
18s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 31m
55s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 27m 21s{color}
| {color:red} root generated 188 new + 1299 unchanged - 0 fixed = 1487 total
(was 1299) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 13s{color} | {color:orange} root: The patch generated 1 new + 1 unchanged -
0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
15s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
38s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HADOOP-15217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12926328/HADOOP-15217.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5c98631d69fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9efb4b7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| compile |
[jira] [Updated] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[ https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zsolt Venczel updated HADOOP-15217: --- Attachment: HADOOP-15217.04.patch > org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces > -- > > Key: HADOOP-15217 > URL: https://issues.apache.org/jira/browse/HADOOP-15217 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Joseph Fourny >Assignee: Zsolt Venczel >Priority: Major > Attachments: HADOOP-15217.01.patch, HADOOP-15217.02.patch, > HADOOP-15217.03.patch, HADOOP-15217.04.patch, TestCase.java > > > When _FsUrlStreamHandlerFactory_ is registered with _java.net.URL_ (ex: when > Spark is initialized), it breaks URLs with spaces (even though they are > properly URI-encoded). I traced the problem down to > _FSUrlConnection.connect()_ method. It naively gets the path from the URL, > which contains encoded spaces, and pases it to > _org.apache.hadoop.fs.Path(String)_ constructor. This is not correct, because > the docs clearly say that the string must NOT be encoded. Doing so causes > double encoding within the Path class (ie: %20 becomes %2520). > See attached JUnit test. > This test case mimics an issue I ran into when trying to use Commons > Configuration 1.9 AFTER initializing Spark. Commons Configuration uses URL > class to load configuration files, but Spark installs > _FsUrlStreamHandlerFactory_, which hits this issue. For now, we are using an > AspectJ aspect to "patch" the bytecode at load time to work-around the issue. > The real fix is quite simple. All you need to do is replace this line in > _org.apache.hadoop.fs.FsUrlConnection.connect()_: > is = fs.open(new Path(url.getPath())); > with this line: > is = fs.open(new Path(url.*toUri()*.getPath())); > URI.getPath() will correctly decode the path, which is what is expected by > _org.apache.hadoop.fs.Path(String)_ constructor. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15217) org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces
[ https://issues.apache.org/jira/browse/HADOOP-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499923#comment-16499923 ] Zsolt Venczel commented on HADOOP-15217: Thanks [~xiaochen] for sharing your thoughts in this matter. >From a code readability and issue debugging perspective I completely agree >with you and I think the problem is twofold: 1) How readable the code is and how much help we provide for the developer while debugging the code. 2) How easy it is to triage a unit test failure which is quite important from a code maintainability perspective on the long run: if I see a unit test producing an error I can think of many things but if it explicitly fails I can assume more safely that some recent change introduced a regression. >From the second perspective the response to your points: ??This is wrapping a code that's supposed to pass. What's the boundary of wrapping this line v.s. wrapping some other lines??? The limit should be as narrow to the tested code as possible. Wrapping more lines would not be meaningful from the actual tests perspective. ??Wrapping here in the test doesn't give more information. My assumption is a developer seeing the IOE from myUrl2.openStream(); would know this means that the openStream failed.?? I was trying to defined the test case to fail on openStream only if the fix is missing and if I managed to set it up correctly it fails only in case of a true regression. Nothing is perfect though, I agree. ??By Assert.fail with ex.getMessage(), we actually lose some information, which is the stacktrace of the IOE. Without the assert, the IOE will fail the test case, presenting us both its message and its stacktrace?? This is a really good point. I updated the patch to include the cause of the exception. ??Let's assume there is an IOE from this in the future. It feels to me it won't be too hard to figure out what's been tested here (since there are code comments, and one can also git blame).?? Currently in the hadoop code base quite a lot of unit tests are failing at each test suite run. If it's an error one might think it's flaky and treat it as so. If it's a failure that feels to be more intentional and one might be more careful and look for a regression. Please let me know if this makes any sense but I can go happily by your suggestions as well. Cheers, Zsolt > org.apache.hadoop.fs.FsUrlConnection does not handle paths with spaces > -- > > Key: HADOOP-15217 > URL: https://issues.apache.org/jira/browse/HADOOP-15217 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 3.0.0 >Reporter: Joseph Fourny >Assignee: Zsolt Venczel >Priority: Major > Attachments: HADOOP-15217.01.patch, HADOOP-15217.02.patch, > HADOOP-15217.03.patch, HADOOP-15217.04.patch, TestCase.java > > > When _FsUrlStreamHandlerFactory_ is registered with _java.net.URL_ (ex: when > Spark is initialized), it breaks URLs with spaces (even though they are > properly URI-encoded). I traced the problem down to > _FSUrlConnection.connect()_ method. It naively gets the path from the URL, > which contains encoded spaces, and pases it to > _org.apache.hadoop.fs.Path(String)_ constructor. This is not correct, because > the docs clearly say that the string must NOT be encoded. Doing so causes > double encoding within the Path class (ie: %20 becomes %2520). > See attached JUnit test. > This test case mimics an issue I ran into when trying to use Commons > Configuration 1.9 AFTER initializing Spark. Commons Configuration uses URL > class to load configuration files, but Spark installs > _FsUrlStreamHandlerFactory_, which hits this issue. For now, we are using an > AspectJ aspect to "patch" the bytecode at load time to work-around the issue. > The real fix is quite simple. All you need to do is replace this line in > _org.apache.hadoop.fs.FsUrlConnection.connect()_: > is = fs.open(new Path(url.getPath())); > with this line: > is = fs.open(new Path(url.*toUri()*.getPath())); > URI.getPath() will correctly decode the path, which is what is expected by > _org.apache.hadoop.fs.Path(String)_ constructor. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
