[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410885#comment-16410885
]
genericqa commented on HDFS-12512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 47s{color} | {color:orange} hadoop-hdfs-project: The patch generated 14 new
+ 125 unchanged - 8 fixed = 139 total (was 133) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 30s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 46s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
31s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.TestMaintenanceState |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-12512 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915807/HDFS-12512.013.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 8816116376
[jira] [Updated] (HDFS-13291) RBF: Implement available space based OrderResolver
[ https://issues.apache.org/jira/browse/HDFS-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-13291: - Attachment: HDFS-13291.006.patch > RBF: Implement available space based OrderResolver > -- > > Key: HDFS-13291 > URL: https://issues.apache.org/jira/browse/HDFS-13291 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Major > Attachments: HDFS-13291.001.patch, HDFS-13291.002.patch, > HDFS-13291.003.patch, HDFS-13291.004.patch, HDFS-13291.005.patch, > HDFS-13291.006.patch > > > Implement available space based OrderResolver, this type resolver will > benefit for balancing the data across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13291) RBF: Implement available space based OrderResolver
[
https://issues.apache.org/jira/browse/HDFS-13291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410857#comment-16410857
]
Yiqun Lin commented on HDFS-13291:
--
Thanks for the review, [~elgoiri] and [~ywskycn]. Attach the updated patch
addressed all comments.
Additionally do one change: Combine the update-period setting and not use two
different key for {{AvailableSpaceResolver}} and {{LocalResolver}}.
> RBF: Implement available space based OrderResolver
> --
>
> Key: HDFS-13291
> URL: https://issues.apache.org/jira/browse/HDFS-13291
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13291.001.patch, HDFS-13291.002.patch,
> HDFS-13291.003.patch, HDFS-13291.004.patch, HDFS-13291.005.patch
>
>
> Implement available space based OrderResolver, this type resolver will
> benefit for balancing the data across subclusters.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[
https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410854#comment-16410854
]
genericqa commented on HDFS-13056:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
40s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 29m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 29m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} root: The patch generated 0 new + 609 unchanged - 1
fixed = 609 total (was 610) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 35s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
33s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}110m 43s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}255m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.util.TestDiskChecker |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/N
[jira] [Updated] (HDFS-13328) Abstract ReencryptionHandler recursive logic in separate class.
[ https://issues.apache.org/jira/browse/HDFS-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-13328: -- Status: Patch Available (was: Open) > Abstract ReencryptionHandler recursive logic in separate class. > --- > > Key: HDFS-13328 > URL: https://issues.apache.org/jira/browse/HDFS-13328 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Surendra Singh Lilhore >Assignee: Surendra Singh Lilhore >Priority: Major > Attachments: HDFS-13328-01.patch > > > HDFS-10899 added DFS logic to scan a directory. It is good to abstract this > logic in separate class, so it can be used in some other feature like > SPS(HDFS-10285). I already tried abstracting DFS logic in HDFS-12291 and same > can be pushed in trunk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13338) Native Build failed in hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410742#comment-16410742
]
Takanobu Asanuma commented on HDFS-13338:
-
Hi [~James C], this may relate to HDFS-8707. Could you check it?
> Native Build failed in hadoop-hdfs-native-client
>
>
> Key: HDFS-13338
> URL: https://issues.apache.org/jira/browse/HDFS-13338
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, native
>Reporter: Takanobu Asanuma
>Priority: Critical
>
> mvn --projects hadoop-hdfs-project/hadoop-hdfs-native-client clean package
> -Pdist,native -DskipTests -Dtar
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part ... dir="/.../hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target"
> executable="cmake">... @ 5:119 in
> /.../hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> around Ant part ... dir="/root/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target"
> executable="cmake">... @ 5:119 in
> /root/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:213)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13338) Native Build failed in hadoop-hdfs-native-client
[
https://issues.apache.org/jira/browse/HDFS-13338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-13338:
Priority: Critical (was: Blocker)
> Native Build failed in hadoop-hdfs-native-client
>
>
> Key: HDFS-13338
> URL: https://issues.apache.org/jira/browse/HDFS-13338
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, native
>Reporter: Takanobu Asanuma
>Priority: Critical
>
> mvn --projects hadoop-hdfs-project/hadoop-hdfs-native-client clean package
> -Pdist,native -DskipTests -Dtar
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part ... dir="/.../hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target"
> executable="cmake">... @ 5:119 in
> /.../hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
> [ERROR] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
> hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
> around Ant part ... dir="/root/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target"
> executable="cmake">... @ 5:119 in
> /root/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
> at org.apache.maven.lifecycle.internal.MojoExecutor.execute
> (MojoExecutor.java:213)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-13250: - Fix Version/s: 3.1.1 > RBF: Router to manage requests across multiple subclusters > -- > > Key: HDFS-13250 > URL: https://issues.apache.org/jira/browse/HDFS-13250 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Fix For: 2.10.0, 2.9.1, 3.0.2, 3.2.0, 3.1.1 > > Attachments: HDFS-13250.000-addendum-branch-2.patch, > HDFS-13250.000.patch, HDFS-13250.001.patch, HDFS-13250.002.patch, > HDFS-13250.003.patch, HDFS-13250.004.patch, HDFS-13250.005.patch > > > HDFS-13124 introduces the concept of mount points spanning multiple > subclusters. The Router should distribute the requests across these > subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13338) Native Build failed in hadoop-hdfs-native-client
Takanobu Asanuma created HDFS-13338:
---
Summary: Native Build failed in hadoop-hdfs-native-client
Key: HDFS-13338
URL: https://issues.apache.org/jira/browse/HDFS-13338
Project: Hadoop HDFS
Issue Type: Bug
Components: build, native
Reporter: Takanobu Asanuma
mvn --projects hadoop-hdfs-project/hadoop-hdfs-native-client clean package
-Pdist,native -DskipTests -Dtar
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
[ERROR] around Ant part .. @ 5:119 in
/.../hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
[ERROR] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal
org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project
hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned: 1
around Ant part .. @ 5:119 in
/root/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/target/antrun/build-main.xml
at org.apache.maven.lifecycle.internal.MojoExecutor.execute
(MojoExecutor.java:213)
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei Yan updated HDFS-12512:
---
Attachment: HDFS-12512.013.patch
> RBF: Add WebHDFS
>
>
> Key: HDFS-12512
> URL: https://issues.apache.org/jira/browse/HDFS-12512
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: fs
>Reporter: Íñigo Goiri
>Assignee: Wei Yan
>Priority: Major
> Labels: RBF
> Attachments: HDFS-12512.000.patch, HDFS-12512.001.patch,
> HDFS-12512.002.patch, HDFS-12512.003.patch, HDFS-12512.004.patch,
> HDFS-12512.005.patch, HDFS-12512.006.patch, HDFS-12512.007.patch,
> HDFS-12512.008.patch, HDFS-12512.009.patch, HDFS-12512.010.patch,
> HDFS-12512.011.patch, HDFS-12512.012.patch, HDFS-12512.013.patch
>
>
> The Router currently does not support WebHDFS. It needs to implement
> something similar to {{NamenodeWebHdfsMethods}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[
https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410720#comment-16410720
]
genericqa commented on HDFS-13277:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12996 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
56s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} HDFS-12996 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m
21s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}139m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13277 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915787/HDFS-13277-HDFS-12996.04.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d4f79823deb1 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12996 / 6c9af6c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23635/testReport/ |
| Max. process+thread count | 3816 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23635/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-13335) Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact
[
https://issues.apache.org/jira/browse/HDFS-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410714#comment-16410714
]
genericqa commented on HDFS-13335:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
28s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
12s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
77m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
24s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 24m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 12s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 19s{color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 20s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:78d94ac |
| JIRA Issue | HDFS-13335 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915778/HDFS-13335-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 307806b60f0d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / 7733a40 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23634/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23634/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/23634/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 1508 (vs. ulimit of 1) |
| modules | C: hadoop-assemblies . U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23634/console |
| Powered by | Apac
[jira] [Commented] (HDFS-13300) Ozone: Remove DatanodeID dependency from HDSL and Ozone
[
https://issues.apache.org/jira/browse/HDFS-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410689#comment-16410689
]
genericqa commented on HDFS-13300:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 45 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
50s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m
40s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
23s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} integration-test in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
27s{color} | {color:red} objectstore-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} ozone-manager in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
24s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-dist hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} hadoop-hdsl/common in HDFS-7240 has 1 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
24s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
25s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} objectstore-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
23s{color} | {color:red} ozone-manager in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
21s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} client in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
21s{color} | {color:red} container-service in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
23s{color} | {color:red} server-scm in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
22s{color} | {color:red} tools in HDFS-7240 failed. {color} |
| {color:red}-1{color} | {color:red} jav
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[
https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410687#comment-16410687
]
genericqa commented on HDFS-13277:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12996 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 18s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} HDFS-12996 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 44s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 435 unchanged - 0 fixed = 436 total (was 435) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 36s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 49s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}154m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestReencryptionWithKMS |
| | hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13277 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915776/HDFS-13277-HDFS-12996.02.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 08078668e7e9 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-12996 / 6c9af6c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apach
[jira] [Commented] (HDFS-12512) RBF: Add WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-12512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410684#comment-16410684
]
genericqa commented on HDFS-12512:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 16m
33s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 33s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-hdfs-project: The patch generated 14 new
+ 125 unchanged - 8 fixed = 139 total (was 133) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 22s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 95m 40s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 26s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-12512 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915767/HDFS-12512.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux d4fc50b4dc01 4.4.0-116-generic #140-Ubuntu SMP
[jira] [Commented] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[
https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410680#comment-16410680
]
genericqa commented on HDFS-13329:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-12996 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
26s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
43s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
52s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
37s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s{color} | {color:green} HDFS-12996 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} HDFS-12996 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
47s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 47s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 47s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 2s{color} | {color:orange} root: The patch generated 11 new + 118 unchanged
- 890 fixed = 129 total (was 1008) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 2m
31s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
16s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 39s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
23s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.TestDU |
| | hadoop.fs.shell.TestCopyFromLocal |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13329 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915790/HDFS-13329-HDFS-12996.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 2b29f6a6
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Open (was: Patch Available) > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, HDFS-13056.009.patch, > Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf, > hdfs-file-composite-crc32-v2.pdf, hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-4275) MiniDFSCluster-based tests fail on Windows due to failure to delete test namenode directory
[
https://issues.apache.org/jira/browse/HDFS-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410653#comment-16410653
]
Íñigo Goiri commented on HDFS-4275:
---
I think there is value in backporting this.
I just opened a new JIRA for this: HDFS-13337.
> MiniDFSCluster-based tests fail on Windows due to failure to delete test
> namenode directory
> ---
>
> Key: HDFS-4275
> URL: https://issues.apache.org/jira/browse/HDFS-4275
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: trunk-win, 3.0.0-alpha1
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-4275.1.patch
>
>
> Multiple HDFS test suites fail on Windows during initialization of
> {{MiniDFSCluster}} due to "Could not fully delete" the name testing data
> directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Status: Patch Available (was: Open) > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, HDFS-13056.009.patch, > Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf, > hdfs-file-composite-crc32-v2.pdf, hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13337) Backport HDFS-4275 to branch-2.9
[ https://issues.apache.org/jira/browse/HDFS-13337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13337: --- Description: Multiple HDFS test suites fail on Windows during initialization of MiniDFSCluster due to "Could not fully delete" the name testing data directory. > Backport HDFS-4275 to branch-2.9 > > > Key: HDFS-13337 > URL: https://issues.apache.org/jira/browse/HDFS-13337 > Project: Hadoop HDFS > Issue Type: Test >Reporter: Íñigo Goiri >Assignee: Xiao Liang >Priority: Minor > > Multiple HDFS test suites fail on Windows during initialization of > MiniDFSCluster due to "Could not fully delete" the name testing data > directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Huo updated HDFS-13056: -- Attachment: HDFS-13056.009.patch > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, HDFS-13056.009.patch, > Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf, > hdfs-file-composite-crc32-v2.pdf, hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[
https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410650#comment-16410650
]
Dennis Huo commented on HDFS-13056:
---
Thanks for the review [~xiaochen]! Responses:
- What's the plan for webhdfs? It seems {{DFSClient#getFileChecksum}} is only
necessary because of webhdfs.
I figured we might want to keep DFSClient#getFileChecksum around in case some
users are calling directly into the java-public interfaces of DFSClient despite
the class being annotated as private, since it's one of the helper classes
close enough to the real public interfaces that it's more likely than, say, the
FileChecksumHelper to have made it into some people's code.
But that was just because I'm not too familiar with the overall policies on
breaking java-public interfaces for classes that are marked with private
InterfaceAudience. I'll be happy to try to remove the MD5-specific
getFileChecksum method entirely from DFSClient if you think it's safe.
It looks like a few files would need to be touched to finish plumbing the
MD5MD5CRC32FileChecksum assumption out, including JsonUtil; let me know if you
think it's better to go ahead and finish the WebHDFS plumbing in this patch, or
to do it in a separate Jira as a followup.
- {{InterfaceAudience.Private}} is by default {{InterfaceStability.Unstable}},
so we can probably remove {{Evolving}} unless there is a reason. We should mark
the {{LimitedPrivate}} classes to be unstable instead of evolving too.
Done
- The {{StripedBlockChecksumReconstructor}} refactor into the
prepareDigester/updateDigester/commitDigest looks a little odd to me. But I
understand if not doing it this way, we'd need to implement 2 mostly-duplicated
{{reconstruct}}. At the minimum we should check for {{digester != null}} in the
latter 2 methods.
Done
- CrcUtil: I see {{writeInt}} and {{readInt}} mentions the requirement about
buffer length v.s. offset, which is good. But for code safety, let's add
validation to actually check the input parameters.
Done
- CrcUtil: newSingleCrcWrapperFromByteArray: how about naming it to something
like toSingleCrcString and a {{String}} instead of an {{Object}}? In the caller
we can also save it to a String variable (and similarly save md5.toString to
it, within an isDebugEnabled check).
Done. The original purpose was to keep it more analogous to the md5 case which
relies on the late evaluation of debug params and thus avoid having to check
isDebugEnabled, but as you noted, I added the isDebugEnabled checks anyways in
response to previous review comments in this Jira
([https://github.com/apache/hadoop/pull/344/commits/1bb2260805c4ecdf014422d2d0eebafe81973a53)],
and I agree it's a bit clunky to have this "Object" wrapper hack. Went ahead
and switched to returning Strings directly since it's easier to read.
- FileChecksumHelper: {{makeFinalResult}} returns null instead of throw when
{{combineMode}} is unknown, unlike other places. I think throw is the right
thing to do.
Done
- FileChecksumHelper: {{tryDatanode}} Let's also log the {{blockChecksumType}}
in the debug log in the end of the method
Done
- BlockChecksumOptions: trivial, more canonical to call the 2-param ctor from
the 1-param ctor, like {{this(blockChecksumType, 0);}}
Done
- LOG.isDebugEnabled is not required if using parameterized messages:
[https://www.slf4j.org/faq.html#logging_performance] So can remove from
{{BlockChecksumHelper$BlockGroupNonStripedChecksumComputer#reassembleNonStripedCompositeCrc}}
Since I switched to CrcUtil.toSingleCrcString as suggested, I kept the
isDebugEnabled checks around the calls instead of relying on late evaluation of
stringifying the crcs.
> Expose file-level composite CRCs in HDFS which are comparable across
> different instances/layouts
>
>
> Key: HDFS-13056
> URL: https://issues.apache.org/jira/browse/HDFS-13056
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, distcp, erasure-coding, federation, hdfs
>Affects Versions: 3.0.0
>Reporter: Dennis Huo
>Assignee: Dennis Huo
>Priority: Major
> Attachments: HDFS-13056-branch-2.8.001.patch,
> HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch,
> HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch,
> HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch,
> HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch,
> HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch,
> HDFS-13056.008.patch, HDFS-13056.009.patch,
> Reference_only_zhen_PPOC_hadoop2.6.X.diff, hdfs-file-composite-crc32-v1.pdf,
> hdfs-file-composite-crc32-v2.pdf, hdfs-file-composite-crc32-v3.pdf
>
>
>
[jira] [Created] (HDFS-13337) Backport HDFS-4275 to branch-2.9
Íñigo Goiri created HDFS-13337: -- Summary: Backport HDFS-4275 to branch-2.9 Key: HDFS-13337 URL: https://issues.apache.org/jira/browse/HDFS-13337 Project: Hadoop HDFS Issue Type: Test Reporter: Íñigo Goiri Assignee: Xiao Liang -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13314) NameNode should optionally exit if it detects FsImage corruption
[
https://issues.apache.org/jira/browse/HDFS-13314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410615#comment-16410615
]
genericqa commented on HDFS-13314:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 24m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 86 unchanged - 1 fixed = 86 total (was 87) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}100m 57s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}179m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13314 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915758/HDFS-13314.05.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d7832a5bf337 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8620d2b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23630/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23630/testReport/ |
| Max. process+thread count | 3838 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-proj
[jira] [Updated] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[ https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13329: -- Status: Patch Available (was: In Progress) > Add/ Update disk space counters for trash (trash used, disk remaining etc.) > > > Key: HDFS-13329 > URL: https://issues.apache.org/jira/browse/HDFS-13329 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13329-HDFS-12996.01.patch > > > Add 3 more counters required for datanode replica trash. > # diskAvailable > # replicaTrashUsed > # replicaTrashRemaining > For more info on these counters, refer design document uploaded in HDFS-12996 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[ https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410599#comment-16410599 ] Bharat Viswanadham commented on HDFS-13329: --- This patch is dependant on HDFS-13277. Adding initial patch. > Add/ Update disk space counters for trash (trash used, disk remaining etc.) > > > Key: HDFS-13329 > URL: https://issues.apache.org/jira/browse/HDFS-13329 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13329-HDFS-12996.01.patch > > > Add 3 more counters required for datanode replica trash. > # diskAvailable > # replicaTrashUsed > # replicaTrashRemaining > For more info on these counters, refer design document uploaded in HDFS-12996 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[ https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13329: -- Attachment: HDFS-13329-HDFS-12996.01.patch > Add/ Update disk space counters for trash (trash used, disk remaining etc.) > > > Key: HDFS-13329 > URL: https://issues.apache.org/jira/browse/HDFS-13329 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13329-HDFS-12996.01.patch > > > Add 3 more counters required for datanode replica trash. > # diskAvailable > # replicaTrashUsed > # replicaTrashRemaining > For more info on these counters, refer design document uploaded in HDFS-12996 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13336) Test cases of TestWriteToReplica failed in windows
[ https://issues.apache.org/jira/browse/HDFS-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Liang updated HDFS-13336: -- Labels: windows (was: ) > Test cases of TestWriteToReplica failed in windows > -- > > Key: HDFS-13336 > URL: https://issues.apache.org/jira/browse/HDFS-13336 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > > Test cases of TestWriteToReplica failed in windows with errors like: > h4. > !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! > Error Details > Could not fully delete > F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 > h4. > !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! > Stack Trace > java.io.IOException: Could not fully delete > F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1011) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:932) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:864) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:497) at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:456) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica.testAppend(TestWriteToReplica.java:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at > org.junit.runners.ParentRunner.run(ParentRunner.java:309) at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13336) Test cases of TestWriteToReplica failed in windows
[ https://issues.apache.org/jira/browse/HDFS-13336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Liang updated HDFS-13336: -- Tags: (was: windows) > Test cases of TestWriteToReplica failed in windows > -- > > Key: HDFS-13336 > URL: https://issues.apache.org/jira/browse/HDFS-13336 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Labels: windows > > Test cases of TestWriteToReplica failed in windows with errors like: > h4. > !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! > Error Details > Could not fully delete > F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 > h4. > !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! > Stack Trace > java.io.IOException: Could not fully delete > F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1011) > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:932) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:864) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:497) at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:456) > at > org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica.testAppend(TestWriteToReplica.java:89) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at > org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at > org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at > org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at > org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at > org.junit.runners.ParentRunner.run(ParentRunner.java:309) at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13336) Test cases of TestWriteToReplica failed in windows
Xiao Liang created HDFS-13336: - Summary: Test cases of TestWriteToReplica failed in windows Key: HDFS-13336 URL: https://issues.apache.org/jira/browse/HDFS-13336 Project: Hadoop HDFS Issue Type: Bug Reporter: Xiao Liang Assignee: Xiao Liang Test cases of TestWriteToReplica failed in windows with errors like: h4. !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! Error Details Could not fully delete F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 h4. !https://builds.apache.org/static/fc5100d0/images/16x16/document_delete.png! Stack Trace java.io.IOException: Could not fully delete F:\short\hadoop-trunk-win\s\hadoop-hdfs-project\hadoop-hdfs\target\test\data\1\dfs\name-0-1 at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1011) at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:932) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:864) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:497) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:456) at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.TestWriteToReplica.testAppend(TestWriteToReplica.java:89) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12884) BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo
[
https://issues.apache.org/jira/browse/HDFS-12884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410579#comment-16410579
]
Konstantin Shvachko commented on HDFS-12884:
Thanks for correcting [~leftnoteasy]. Missed that one.
> BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo
> ---
>
> Key: HDFS-12884
> URL: https://issues.apache.org/jira/browse/HDFS-12884
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.7.4
>Reporter: Konstantin Shvachko
>Assignee: chencan
>Priority: Major
> Fix For: 2.10.0, 2.9.1, 2.8.4, 2.7.6, 3.0.2, 3.2.0, 3.1.1
>
> Attachments: HDFS-12884.001.patch, HDFS-12884.002.patch,
> HDFS-12884.003.patch
>
>
> {{BlockUnderConstructionFeature.truncateBlock}} type should be changed to
> {{BlockInfo}} from {{Block}}. {{truncateBlock}} is always assigned as
> {{BlockInfo}}, so this will avoid unnecessary casts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410574#comment-16410574 ] Bharat Viswanadham commented on HDFS-13277: --- Missed javadoc for few. Addressed them in patch v04. > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch, > HDFS-13277-HDFS-12996.03.patch, HDFS-13277-HDFS-12996.04.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13277: -- Attachment: HDFS-13277-HDFS-12996.04.patch > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch, > HDFS-13277-HDFS-12996.03.patch, HDFS-13277-HDFS-12996.04.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410572#comment-16410572 ] Bharat Viswanadham commented on HDFS-13277: --- Attached patch v03. Thank You [~ajayydv] for review, addressed review comments. > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch, > HDFS-13277-HDFS-12996.03.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13277: -- Attachment: HDFS-13277-HDFS-12996.03.patch > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch, > HDFS-13277-HDFS-12996.03.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410549#comment-16410549 ] Ajay Kumar commented on HDFS-13277: --- [~bharatviswa] thanks for working on this. Patch 2 looks good. Some suggestions: * FsDatasetAsyncDiskService L346: Do we need this else if check as previous else checks if it is equal. * javadoc for ReplicaTrashInfo & FsDatasetAsyncDiskService#replicaTrashInfoMap * Rename ReplicaTrashInfo to ReplicaTrashCurDirInfo ? * FsDatasetAsyncDiskService L102: rename replicaTrashSubDirMax to replicaTrashSubDirMaxBlocks or replicaTrashSubDirMaxFiles ? > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13300) Ozone: Remove DatanodeID dependency from HDSL and Ozone
[
https://issues.apache.org/jira/browse/HDFS-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410541#comment-16410541
]
Nanda kumar commented on HDFS-13300:
Thanks [~xyao] and [~elek] for the review, updated the patch v003 accordingly.
Please find my response below
Hdsl.proto
{quote}should we make ipAddress/hostName/infoPort required?
{quote}
Made ipAddress/hostName as a required field. infoPort is not required in
HDSL/Ozone, HDFS-13324 is to track the removal of InfoPort and InfoSecurePort.
StorageContainerDatanodeProtocol.proto
{quote}Line 35-40: unused imports can be removed from proto file (IntelliJ
won't report unused protobuf import)
{quote}
Fixed
CommandQueue.java
{quote}DatanodeDetails.Uuid is better than DatanodeDetails.datanodeUuid
{quote}
Fixed
ContainerManagerImpl.java
{quote}Line 124: NIT: remove the "ID"
{quote}
Fixed
DatanodeDeletedBlockTransactions.java
Line 44: this map can be keyed by UUID
Fixed
DatanodeDetails.java
{quote}Can we directly use the protobuf generated class to avoid conversions?
{quote}
This can be tricky as we made IP and hostname a required field in protobuf.
When we create DatanodeDetails instance in {{HdslDatanodeService}} we don't
have value for IP/hostname, it's set later by XceiverServer/XceiverServerRatis.
If we directly use protobuf generated class we have to set the values in
builder before calling {{builder.build()}}
HdfsUtils.java
{quote}Unrelated change in hdfs can be avoided.
{quote}
Fixed
HdslDatanodeService.java
{quote}Line 80: can we define a separate SCM Exception for this?
{quote}
{{IllegalArgumentException}} is used in a similar manner in HDSL/Ozone whenever
a mandatory property is missing. Created HDFS-1 to introduce new Exception
to handle such cases.
DataNodeServicePlugin.java
{quote}Agree with @Marton that we should use the ServicePlugin directly.
{quote}
Planning to handle this as part of HDFS-13325
InitDatanodeState.java
{quote}Line 109: agree with @Marton that we might not need to persist all as
the port may change or unavailable across a restart.
{quote}
Created HDFS-13334 to track this
ObjectStoreRestPlugin.java
{quote}Line 78-86: can be simplified with super class's implementation of the
same method.
{quote}
The super class/interface (DataNodeServicePlugin) will be removed as part of
HDFS-13325
SCMNodeManager.java
{quote}Line 126: can we change the nodeStats map to be keyed by UUID as well?
{quote}
Fixed
{quote}ServiceDiscovery is based on the healthyNodes/staleNodes/deadNodes maps
in SCMNodeManager. Let me know if I am wrong but I can imagine the following
situation.
# HdslDatanodeService.start() is called by the DataNode()
## DatanodeDescriptor is created
## DatanodeMachine is started (Async! Separated threads)
# ObjectStoreRestPlugin.start() is called
## In the mean time the DatanodeMachine is started in a separated thread (see
the previous point). RunningDatanodeState may send the first registration
request to the SCM.
## Rest interface is started and restPort is updated in the
DatanodeDescriptor. But the descriptor has already been sent to the SCM.
# With the next heartbeats the latest datanodeDescriptor will be sent to the
scm. But the maps (healthyNodes/staleNodes/deadNodes) are not updated with the
new descriptor. Still the original descriptor will be used which (in some
cases) doesn't contain the rest port.{quote}
@Marton, your understanding is completely right, but the ports are set during
the construction of {{DatanodeStateMachine}} not after
{{datanodeStateMachine.startDaemon()}}. So when
{{ObjectStoreRestPlugin.start()}} we have already created
{{DatanodeStateMachine}} instance which would have updated the ports properly.
In DataNode#startPlugins
# HdslDatanodeService#start is called
## Create DatanodeDetails --> Blocking call
## Create DatanodeStateMachine --> Blocking call
### Create OzoneContainer instance --> Blocking call
Create XceiverServer --> Blocking call
# Port is updated in DatanodeDetails reference
Create XceiverServerRatis --> Blocking call
# Port is updated in DatanodeDetails reference
## Start DatanodeStateMachine --> NON Blocking call (Control is returned to
DataNode)
# ObjectStoreRestPlugin.start is called (The ports are already updated through
DatanodeStateMachine)
> Ozone: Remove DatanodeID dependency from HDSL and Ozone
>
>
> Key: HDFS-13300
> URL: https://issues.apache.org/jira/browse/HDFS-13300
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Major
> Attachments: HDFS-13300-HDFS-7240.000.patch,
> HDFS-13300-HDFS-7240.001.patch, HDFS-13300-HDFS-7240.002.patch,
> HDFS-13300-HDFS-7240.003.patch
>
>
> Datanode
[jira] [Commented] (HDFS-4275) MiniDFSCluster-based tests fail on Windows due to failure to delete test namenode directory
[
https://issues.apache.org/jira/browse/HDFS-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410536#comment-16410536
]
Xiao Liang commented on HDFS-4275:
--
Should backport this fix to branch-2 and branch-2.9 ?
> MiniDFSCluster-based tests fail on Windows due to failure to delete test
> namenode directory
> ---
>
> Key: HDFS-4275
> URL: https://issues.apache.org/jira/browse/HDFS-4275
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: trunk-win, 3.0.0-alpha1
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-4275.1.patch
>
>
> Multiple HDFS test suites fail on Windows during initialization of
> {{MiniDFSCluster}} due to "Could not fully delete" the name testing data
> directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13300) Ozone: Remove DatanodeID dependency from HDSL and Ozone
[ https://issues.apache.org/jira/browse/HDFS-13300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDFS-13300: --- Attachment: HDFS-13300-HDFS-7240.003.patch > Ozone: Remove DatanodeID dependency from HDSL and Ozone > > > Key: HDFS-13300 > URL: https://issues.apache.org/jira/browse/HDFS-13300 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Attachments: HDFS-13300-HDFS-7240.000.patch, > HDFS-13300-HDFS-7240.001.patch, HDFS-13300-HDFS-7240.002.patch, > HDFS-13300-HDFS-7240.003.patch > > > DatanodeID has been modified to add HDSL/Ozone related information > previously. This jira is to remove DatanodeID dependency from HDSL/Ozone to > make it truly pluggable without having the need to modify DatanodeID. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[ https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13329: -- Attachment: (was: HDFS-13329-HDFS-12996.00.patch) > Add/ Update disk space counters for trash (trash used, disk remaining etc.) > > > Key: HDFS-13329 > URL: https://issues.apache.org/jira/browse/HDFS-13329 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > Add 3 more counters required for datanode replica trash. > # diskAvailable > # replicaTrashUsed > # replicaTrashRemaining > For more info on these counters, refer design document uploaded in HDFS-12996 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13335) Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact
[ https://issues.apache.org/jira/browse/HDFS-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13335: Status: Patch Available (was: Open) To test: 1. Do a full build (with -Phdsl) 2. Do a mvn install -Psrc (the src tar file in hadoop-dist/target/ won't include the ozone/dsl files) 3. Do a mvn install -Phdsl-src (additional source tar will be created with hdsl source) > Ozone: remove hdsl/cblock/ozone source code from the official apache source > release artifact > > > Key: HDFS-13335 > URL: https://issues.apache.org/jira/browse/HDFS-13335 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13335-HDFS-7240.001.patch > > > [~andrew.wang] pointed in the [adoption > thread|https://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201803.mbox/%3CCAGB5D2a5qL%3DG266pqWm1_%2ByTvpPyjJ%2B_ebt0uf-bk9LHbnp1TA%40mail.gmail.com%3E] > that the official hadoop source release should not contain cblock/hdsl/ozone > code. > This patch contains a simple adjustment of the assembly descriptor to remove > them. And an additional separated profile (hdsl-src) to create a full-source > tar file (won't be invoked by the create-release script). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13329) Add/ Update disk space counters for trash (trash used, disk remaining etc.)
[ https://issues.apache.org/jira/browse/HDFS-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13329: -- Attachment: HDFS-13329-HDFS-12996.00.patch > Add/ Update disk space counters for trash (trash used, disk remaining etc.) > > > Key: HDFS-13329 > URL: https://issues.apache.org/jira/browse/HDFS-13329 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13329-HDFS-12996.00.patch > > > Add 3 more counters required for datanode replica trash. > # diskAvailable > # replicaTrashUsed > # replicaTrashRemaining > For more info on these counters, refer design document uploaded in HDFS-12996 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13335) Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact
[ https://issues.apache.org/jira/browse/HDFS-13335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDFS-13335: Attachment: HDFS-13335-HDFS-7240.001.patch > Ozone: remove hdsl/cblock/ozone source code from the official apache source > release artifact > > > Key: HDFS-13335 > URL: https://issues.apache.org/jira/browse/HDFS-13335 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Attachments: HDFS-13335-HDFS-7240.001.patch > > > [~andrew.wang] pointed in the [adoption > thread|https://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201803.mbox/%3CCAGB5D2a5qL%3DG266pqWm1_%2ByTvpPyjJ%2B_ebt0uf-bk9LHbnp1TA%40mail.gmail.com%3E] > that the official hadoop source release should not contain cblock/hdsl/ozone > code. > This patch contains a simple adjustment of the assembly descriptor to remove > them. And an additional separated profile (hdsl-src) to create a full-source > tar file (won't be invoked by the create-release script). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13335) Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact
Elek, Marton created HDFS-13335: --- Summary: Ozone: remove hdsl/cblock/ozone source code from the official apache source release artifact Key: HDFS-13335 URL: https://issues.apache.org/jira/browse/HDFS-13335 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Elek, Marton Assignee: Elek, Marton [~andrew.wang] pointed in the [adoption thread|https://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201803.mbox/%3CCAGB5D2a5qL%3DG266pqWm1_%2ByTvpPyjJ%2B_ebt0uf-bk9LHbnp1TA%40mail.gmail.com%3E] that the official hadoop source release should not contain cblock/hdsl/ozone code. This patch contains a simple adjustment of the assembly descriptor to remove them. And an additional separated profile (hdsl-src) to create a full-source tar file (won't be invoked by the create-release script). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13334) Ozone: Store only datanode UUID in datanode.id file
Nanda kumar created HDFS-13334: -- Summary: Ozone: Store only datanode UUID in datanode.id file Key: HDFS-13334 URL: https://issues.apache.org/jira/browse/HDFS-13334 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Nanda kumar Now we are storing complete DatanodeDetailsProto to the datanode.id file, it's sufficient to store just the UUID of datanode as other details might change between restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13277: -- Attachment: HDFS-13277-HDFS-12996.02.patch > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410519#comment-16410519 ] Bharat Viswanadham commented on HDFS-13277: --- Thank You [~hanishakoneru] for review. Addressed review comments and fixed findbug warnings > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13277) Improve move to account for usage (number of files) to limit trash dir size
[ https://issues.apache.org/jira/browse/HDFS-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410519#comment-16410519 ] Bharat Viswanadham edited comment on HDFS-13277 at 3/22/18 10:41 PM: - Attached v02 patch Thank You [~hanishakoneru] for review. Addressed review comments and fixed findbug warnings was (Author: bharatviswa): Thank You [~hanishakoneru] for review. Addressed review comments and fixed findbug warnings > Improve move to account for usage (number of files) to limit trash dir size > --- > > Key: HDFS-13277 > URL: https://issues.apache.org/jira/browse/HDFS-13277 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13277-HDFS-12996.00.patch, > HDFS-13277-HDFS-12996.01.patch, HDFS-13277-HDFS-12996.02.patch > > > The trash subdirectory maximum entries. This puts an upper limit on the size > of subdirectories in replica-trash. Set this default value to > blockinvalidateLimit. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13333) Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing
Nanda kumar created HDFS-1: -- Summary: Ozone: Introduce a new SCM Exception which will be thrown when mandatory property is missing Key: HDFS-1 URL: https://issues.apache.org/jira/browse/HDFS-1 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Nanda kumar It's better to have a separate SCM Exception to indicate a missing mandatory property. This was proposed by [~xyao] in [this comment| https://issues.apache.org/jira/browse/HDFS-13300?focusedCommentId=16408553&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16408553] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13332) Ozone: update log4j.properties changes for hdsl/ozone.
Xiaoyu Yao created HDFS-13332: - Summary: Ozone: update log4j.properties changes for hdsl/ozone. Key: HDFS-13332 URL: https://issues.apache.org/jira/browse/HDFS-13332 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao *hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties* -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13331) Add lastSeenStateId to RpcRequestHeader.
Plamen Jeliazkov created HDFS-13331: --- Summary: Add lastSeenStateId to RpcRequestHeader. Key: HDFS-13331 URL: https://issues.apache.org/jira/browse/HDFS-13331 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov HDFS-12977 added a stateId into the RpcResponseHeader which is returned by NameNode and stored by DFSClient. This JIRA is to followup on that work and have the DFSClient send their stored "lastSeenStateId" in the RpcRequestHeader so that ObserverNodes can then compare with their own and act accordingly. This JIRA work focuses on just the part of making DFSClient send their state through RpcRequestHeader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11518) libhdfs++: Add a build option to skip building examples, tests, and tools
[ https://issues.apache.org/jira/browse/HDFS-11518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410466#comment-16410466 ] Hudson commented on HDFS-11518: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11518: libhdfs++: Add a build option to skip building examples, (james.clampffer: rev 1c9b174663f4f73f7cce275e2d7b302c7d37d293) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt > libhdfs++: Add a build option to skip building examples, tests, and tools > - > > Key: HDFS-11518 > URL: https://issues.apache.org/jira/browse/HDFS-11518 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-11518.HDFS-8707.000.patch, > HDFS-11518.HDFS-8707.001.patch, HDFS-11518.HDFS-8707.002.patch > > > Adding a flag to just build the core library without tools, examples, and > tests will make it easier and lighter weight to embed the libhdfs++ source as > a third-party component of other projects. It won't need to look for a JDK, > valgrind, and gmock and won't generate a handful of binaries that might not > be relevant to other projects during normal use. > This should also make it a bit easier to wire into other build frameworks > since there won't be standalone binaries that need the path to other > libraries like protobuf while the library builds. They just need to be > around while the project embedding libhdfs++ gets linked. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11544) libhdfs++: Improve C API error reporting
[ https://issues.apache.org/jira/browse/HDFS-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410461#comment-16410461 ] Hudson commented on HDFS-11544: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11544: libhdfs++: Improve C API error reporting. Contributed by (james.clampffer: rev fdb88eb57e18273d31681d31202320da0b0ca61e) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc HDFS-11544: libhdfs++: Revert HDFS-11544.HDFS-8707.001.patch due to (james.clampffer: rev 52445fd4ee73df0bd113e257fb4435796241486e) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc > libhdfs++: Improve C API error reporting > > > Key: HDFS-11544 > URL: https://issues.apache.org/jira/browse/HDFS-11544 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11544.HDFS-8707.000.patch, > HDFS-11544.HDFS-8707.001.patch, HDFS-11544.HDFS-8707.002.patch > > > The thread local string used for hdfsGetLastError wasn't reset between calls > so it could give stale results in confusing ways. Now it gets reset with a > placeholder that says that hasn't set an error string. > Also fixed indentation that wasn't consistent + marked which functions are > used by the hdfs.h API and hdfs_ext.h API to make it easier to see when > changes could break compatibility. Included some minor cleanup for the > common case catch blocks. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11028) libhdfs++: FileSystem needs to be able to cancel pending connections
[ https://issues.apache.org/jira/browse/HDFS-11028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410458#comment-16410458 ] Hudson commented on HDFS-11028: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11028: libhdfs++: FileSystem needs to be able to cancel pending (james.clampffer: rev 8783461e2ec3aff2630ea3574a69beb1d5c61e84) * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/connect_cancel/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/continuation.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/connect_cancel.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/connect_cancel/connect_cancel.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/asio.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc > libhdfs++: FileSystem needs to be able to cancel pending connections > > > Key: HDFS-11028 > URL: https://issues.apache.org/jira/browse/HDFS-11028 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11028.HDFS-8707.000.patch, > HDFS-11028.HDFS-8707.001.patch, HDFS-11028.HDFS-8707.002.patch, > HDFS-11028.HDFS-8707.003.patch, HDFS-11028.HDFS-8707.004.patch > > > Cancel support is now reasonably robust except the case where a FileHandle > operation ends up causing the RpcEngine to try to create a new RpcConnection. > In HA configs it's common to have something like 10-20 failovers and a 20 > second failover delay (no exponential backoff just yet). This means that all > of the functions with synchronous interfaces can still block for many minutes > after an operation has been canceled, and often the cause of this is > something trivial like a bad config file. > The current design makes this sort of thing tricky to do because the > FileHandles need to be individually cancelable via CancelOperations, but they > share the RpcEngine that does the async magic. > Updated design: > Original design would end up forcing lots of reconnects. Not a huge issue on > an unauthenticated cluster but on a kerberized cluster this is a recipe for > Kerberos thinking we're attempting a replay attack. > User visible cancellation and internal resources cleanup are separable > issues. The former can be implemented by atomically swapping the callback of > the operation to be canceled with a no-op callback. The original callback is > then posted to the IoService with an OperationCanceled status and the user is > no longer blocked. For RPC cancels this is sufficient, it's not expensive to > keep a request around a little bit longer and when it's eventually invoked or > timed out it invokes the no-op callback and is ignored (other than a trace > level log notification). Connect cancels push a flag down into the RPC > engine to kill the connection an
[jira] [Commented] (HDFS-12134) libhdfs++: Add a synchronization interface for the GSSAPI
[
https://issues.apache.org/jira/browse/HDFS-12134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410474#comment-16410474
]
Hudson commented on HDFS-12134:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13869/])
HDFS-12134: libhdfs++: Add a synchronization interface for the GSSAPI.
(james.clampffer: rev 22ea06a3dd74b80737977908e30ec38a32b968ab)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt
* (add)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/locks.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt
* (add)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/locks.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/uri_test.cc
* (add)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/user_lock_test.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/gsasl_engine.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/uri.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc
> libhdfs++: Add a synchronization interface for the GSSAPI
> -
>
> Key: HDFS-12134
> URL: https://issues.apache.org/jira/browse/HDFS-12134
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-12134.HDFS-8707.000.patch,
> HDFS-12134.HDFS-8707.001.patch, HDFS-12134.HDFS-8707.002.patch,
> HDFS-12134.HDFS-8707.003.patch, HDFS-12134.HDFS-8707.004.patch
>
>
> Bits of the GSSAPI that Cyrus Sasl uses aren't thread safe. There needs to
> be a way for a client application to share a lock with this library in order
> to prevent race conditions. It can be done using event callbacks through the
> C API but we can provide something more robust (RAII) in the C++ API.
> Proposed client supplied lock, pretty much the C++17 lockable concept. Use a
> default if one isn't provided. This would be scoped at the process level
> since it's unlikely that multiple instances of libgssapi unless someone puts
> some effort in with dlopen/dlsym.
> {code}
> class LockProvider
> {
> virtual ~LockProvider() {}
> // allow client application to deny access to the lock
> virtual bool try_lock() = 0;
> virtual void unlock() = 0;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12110) libhdfs++: Rebase 8707 branch onto an up to date version of trunk
[ https://issues.apache.org/jira/browse/HDFS-12110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410479#comment-16410479 ] Hudson commented on HDFS-12110: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12110: libhdfs++: Rebase HDFS-8707 on trunk, minor conflict (james.clampffer: rev 8620d2bdf9ddeaa28fd0f5ddce984c44025e02a7) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfs_wrapper_defines.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfs_wrapper_undefs.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfspp_wrapper_defines.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto/CMakeLists.txt * (edit) dev-support/docker/Dockerfile > libhdfs++: Rebase 8707 branch onto an up to date version of trunk > - > > Key: HDFS-12110 > URL: https://issues.apache.org/jira/browse/HDFS-12110 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Deepak Majeti >Priority: Major > Attachments: HDFS-12110.HDFS-8707.000.patch, HDFS-12110.diff > > > It's been way too long since this has been done and it's time to start > knocking down blockers for merging into trunk. Can most likely just > copy/paste the libhdfs++ directory into a newer version of master. Want to > track it in a jira since it's likely to cause conflicts when pulling the > updated branch for the first time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10787) libhdfs++: Public API should expose configuration parser
[ https://issues.apache.org/jira/browse/HDFS-10787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410476#comment-16410476 ] Hudson commented on HDFS-10787: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10787: libhdfs++: Public API should expose configuration parser. (james.clampffer: rev 12942f679ae86720c79df033506c53dc1145aa91) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/find/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/protobuf.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_configuration_test.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/config_parser.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/cat.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/gendirs/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/config_parser.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/gendirs/gendirs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/find/find.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/tools_common.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/tools_common.cc > libhdfs++: Public API should expose configuration parser > > > Key: HDFS-10787 > URL: https://issues.apache.org/jira/browse/HDFS-10787 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10787.HDFS-8707.000.patch, > HDFS-10787.HDFS-8707.001.patch, HDFS-10787.HDFS-8707.002.patch, > HDFS-10787.HDFS-8707.003.patch > > > Currently, libhdfspp examples and tools all have this: > #include "hdfspp/hdfspp.h" > #include "common/hdfs_configuration.h" > #include "common/configuration_loader.h" > This is done in order to read configs and connect. We want > hdfs_configuration and configuration_loader to be accessible just by > including our hdfspp.h. One way to achieve that would be to create a builder, > and would include the above libraries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --
[jira] [Commented] (HDFS-11294) libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails
[ https://issues.apache.org/jira/browse/HDFS-11294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410477#comment-16410477 ] Hudson commented on HDFS-11294: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11294: libhdfs++: Segfault in HA failover if DNS lookup for both (james.clampffer: rev 48db24a430f5911a564d08f9ae927aaa9d81cd50) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/namenode_tracker.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_config_connect_bugs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/events.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/namenode_tracker.cc > libhdfs++: Segfault in HA failover if DNS lookup for both Namenodes fails > - > > Key: HDFS-11294 > URL: https://issues.apache.org/jira/browse/HDFS-11294 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11294.HDFS-8707.001.patch, > HDFS-11294.HDFS-8707.002.patch, HDFS-8707.HDFS-11294.000.patch > > > Hit while doing more manual testing on HDFS-11028. > The HANamenodeTracker takes an asio endpoint to figure out what endpoint on > the other node to try next during a failover. This is done by passing the > element at index 0 from endpoints (a std::vector). > When DNS fails the endpoints vector for that node will be empty so the > iterator returned by endpoints\[0\] is just a null pointer that gets > dereferenced and causes a segfault. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11971) libhdfs++: A few portability issues
[
https://issues.apache.org/jira/browse/HDFS-11971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410467#comment-16410467
]
Hudson commented on HDFS-11971:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13869/])
HDFS-11971: libhdfs++: [Fix] a few portability issues. Contributed by
(james.clampffer: rev c5e7a69523f4bce0f09c7afe423452c36213a072)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/find/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/gendirs/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/connect_cancel/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/CMakeLists.txt
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/connect_cancel/CMakeLists.txt
> libhdfs++: A few portability issues
> ---
>
> Key: HDFS-11971
> URL: https://issues.apache.org/jira/browse/HDFS-11971
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Anatoli Shein
>Assignee: Anatoli Shein
>Priority: Major
> Attachments: HDFS-11971.HDFS-8707.000.patch,
> HDFS-11971.HDFS-8707.001.patch, HDFS-11971.HDFS-8707.002.patch,
> HDFS-11971.HDFS-8707.003.patch
>
>
> I recently encountered a few portability issues with libhdfs++ while trying
> to build it as a stand alone project (and also as part of another Apache
> project).
> 1. Method fixCase in configuration.h file produces a warning "conversion to
> ‘char’ from ‘int’ may alter its value [-Werror=conversion]" which does not
> allow libhdfs++ to be compiled as part of the codebase that treats such
> warnings as errors (can be fixed with a simple cast).
> 2. In CMakeLists.txt file (in libhdfspp directory) we do
> find_package(Threads) however we do not link it to the targets (e.g.
> hdfspp_static), which causes the build to fail with pthread errors. After the
> Threads package is found we need to link it using ${CMAKE_THREAD_LIBS_INIT}.
> 3. All the tools and examples fail to build as part of a standalone libhdfs++
> because they are missing multiple libraries such as protobuf, ssl, pthread,
> etc. This happens because we link them to a shared library hdfspp instead of
> hdfspp_static library. We should either link all the tools and examples to
> hdfspp_static library or explicitly add linking to all missing libraries for
> each tool/example.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12427) libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine
[ https://issues.apache.org/jira/browse/HDFS-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410478#comment-16410478 ] Hudson commented on HDFS-12427: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12427: libhdfs++: Prevent requests from holding dangling pointer to (james.clampffer: rev fc5e44d9edeaad9c1381a46fb370b167ac6250b1) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto/protoc_gen_hrpc.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/request.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/request.cc > libhdfs++: Prevent Requests from holding dangling pointer to RpcEngine > -- > > Key: HDFS-12427 > URL: https://issues.apache.org/jira/browse/HDFS-12427 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-12427.HDFS-8707.000.patch, > HDFS-12427.HDFS-8707.001.patch, HDFS-12427.HDFS-8707.002.patch, > HDFS-12427.HDFS-8707.003.patch, HDFS-12427.HDFS-8707.004.patch > > > The lifetime of Request objects is tied to the worker thread(s) in the async > event loop. In the current code there's nothing that prevents a request from > outliving the RpcEngine (bound to FileSystem) while it's waiting for IO. If > the Request, or a task that makes a new request, outlives the RpcEngine it > attempts to dereference a dangling pointer and either crashes or continues to > run with bad data. > Proposed fix is to reference count the RpcEngine via shared_ptr so that > Requests can hold a weak_ptr to it. When a request or RpcConnection > attempting to make a request needs something from the RpcEngine like a call > id number it can promote the weak_ptr to a shared_ptr. If it's unable to > promote because the RpcEngine has been destroyed the Request's handler can be > invoked with an appropriate error message. A weak_ptr must be used rather > than a shared_ptr to avoid reference cycles. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10754) libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, hdfs_chown, hdfs_chmod and hdfs_find.
[ https://issues.apache.org/jira/browse/HDFS-10754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410445#comment-16410445 ] Hudson commented on HDFS-10754: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10754: libhdfs++: Create tools directory and implement hdfs_cat, (james.clampffer: rev 4f6cb5d1a150b366516ddfc3409925c26a1a2a6a) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/gendirs/gendirs.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/tools_common.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/find/find.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/find/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfspp_wrapper_defines.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/cat/cat.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chmod.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/tools_common.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_configuration_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chgrp.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chown.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/statinfo.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration_loader.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_cat.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/configuration_test.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfs_wrapper_undefs.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_find.cpp > libhdfs++: Create tools directory and implement hdfs_cat, hdfs_chgrp, > hdfs_chown, hdfs_chmod and hdfs_find. > --- > > Key: HDFS-10754 > URL: https://issues.apache.org/jira/browse/HDFS-10754 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-10754.HDFS-8707.000.patch, > HDFS-10754.HDFS-8707.001.patch, HDFS-10754.HDFS-8707.002.patch, > HDFS-10754.HDFS-8707.003.patch, HDFS-10754.HDFS-8707.004.patch, > HDFS-10754.HDFS-8707.005.patch, HDFS-10754.HDFS-8707.006.patch, > HDFS-10754.HDFS-8707.007.patch, HDFS-10754.HDFS-8707.008.patch, > HDFS-10754.HDFS-8707.009.patch, HDFS-10754.HDFS-8707.010.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e
[jira] [Commented] (HDFS-10672) libhdfs++: reorder directories in src/main/libhdfspp/examples, and add C++ version of cat tool
[ https://issues.apache.org/jira/browse/HDFS-10672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410439#comment-16410439 ] Hudson commented on HDFS-10672: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10672: libhdfs++: reorder directories in (james.clampffer: rev 649aff11fe618fee642fe4e617b94a581602a258) * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat/cat.c * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/cat/cat.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util_c.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/cat/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/CMakeLists.txt * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/CMakeLists.txt * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/bad_datanode_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc > libhdfs++: reorder directories in src/main/libhdfspp/examples, and add C++ > version of cat tool > -- > > Key: HDFS-10672 > URL: https://issues.apache.org/jira/browse/HDFS-10672 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-10672.HDFS-8707.000.patch, > HDFS-10672.HDFS-8707.001.patch, HDFS-10672.HDFS-8707.002.patch, > HDFS-10672.HDFS-8707.003.patch, HDFS-10672.HDFS-8707.004.patch > > > src/main/libhdfspp/examples should be structured like > examples/language/utility instead of examples/utility/language for easier > access by different developers. > Additionally implementing C++ version of cat tool. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12237) libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf library is built from source
[ https://issues.apache.org/jira/browse/HDFS-12237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410475#comment-16410475 ] Hudson commented on HDFS-12237: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12237: libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf (james.clampffer: rev eeb49d0ca7149b4dd4204f952983cdc6f0471f00) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt > libhdfs++: PROTOC_IS_COMPATIBLE check fails if protobuf library is built from > source > > > Key: HDFS-12237 > URL: https://issues.apache.org/jira/browse/HDFS-12237 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-12237.HDFS-8707.000.patch, > HDFS-12237.HDFS-8707.001.patch, HDFS-12237.HDFS-8707.002.patch > > > Looks like the PROTOC_IS_COMPATIBLE check fails when Protobuf library is > built from source. This happens because the check if performed during the > cmake phase, and the protobuf library needed for this test is build from > source only during the make phase, so the check fails with "ld: cannot find > -lprotobuf" because it was not built yet. We should probably restrict this > test to run only in cases when Protobuf library is already present and not > being built from source. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10796) libhdfs++: rationalize ioservice interactions
[ https://issues.apache.org/jira/browse/HDFS-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410455#comment-16410455 ] Hudson commented on HDFS-10796: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10796: libhdfs++: rationalize ioservice interactions. Contributed (james.clampffer: rev fbff671e0fae04b3ab8e11684de376f9c5c5f74a) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfspp_mini_dfs.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_ioservice.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/options.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_ioservice.cc HDFS-10796: libhdfs++: Previous commit was missing (james.clampffer: rev a30cf6a369770381df4b3fb23d059749dbf6c4cf) * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ioservice_test.cc > libhdfs++: rationalize ioservice interactions > - > > Key: HDFS-10796 > URL: https://issues.apache.org/jira/browse/HDFS-10796 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10796.HDFS-8707.000.patch > > > Firstly, we should be pulling the number of threads from options.io_threads > (which should default to std::thread::hardware_concurrency()). The library > should pass all tests always with io_threads set to 1 or to number> > Secondly, we should have _a_ constructor where the consumer doesn't need to > manage the IOService explicitly, and the FileSystemImpl should create its own > internally. > Since the FileSystem is defined as being for a particular user/identity, > there is a valid use case for the consumer to be constructing many FileSystem > instances to represent many authenticated users in the same process, but want > to share resources (notably have a single io_service shared amongst them > all). In this case, the consumer would want to own the IOService and pass > the same instance to multiple FileSystem instances. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12104) libhdfs++: Make sure all steps in SaslProtocol end up calling AuthComplete
[ https://issues.apache.org/jira/browse/HDFS-12104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410469#comment-16410469 ] Hudson commented on HDFS-12104: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12104: libhdfs++: Make sure all steps in SaslProtocol end up (james.clampffer: rev e64ba87486b9cb40474dfe27cf735cf179cb174c) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc > libhdfs++: Make sure all steps in SaslProtocol end up calling AuthComplete > --- > > Key: HDFS-12104 > URL: https://issues.apache.org/jira/browse/HDFS-12104 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-12104.HDFS-8707.000.patch > > > SaslProtocol provides an abstraction for stepping through the authentication > challenge and response stages in the Cyrus SASL library by chaining callbacks > together (next one is invoked when async io is done). > To authenticate SaslProtocol::Authenticate is called, and when authentication > is finished SaslProtocol::AuthComplete is called which invokes an > authentication completion callback. There's a couple cases where the > intermediate callbacks return without calling AuthComplete which breaks > applications that take advantage of that callback. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11436) libhdfs++: Fix race condition in ScopedResolver
[ https://issues.apache.org/jira/browse/HDFS-11436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410459#comment-16410459 ] Hudson commented on HDFS-11436: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11436: libhdfs++: Fix race condition in ScopedResolver. (james.clampffer: rev 606d20c6c0678be0cc7d0163defceacd1ffe03ae) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.cc > libhdfs++: Fix race condition in ScopedResolver > --- > > Key: HDFS-11436 > URL: https://issues.apache.org/jira/browse/HDFS-11436 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11436.HDFS-8707.000.patch > > > ScopedResolver holds a shared_ptr to a std::promise to do async stuff while > the calling thread blocks. The spec allows promise::set_value to unblock > future::wait immediately which causes future::get_value to race to read > whatever the promise contains before the state and value owned by the promise > is destroyed. In this case even though the ScopedResolver holds a shared_ptr > to the promise which looks like it'd prevent this but it's possible for wait > to unblock fast enough for the destructor on ScopedResolver to be called > which ends up destroying the promise. GCC's implementation of promises and > futures will hit this and will show up in valgrind as a bunch of invalid > reads if the timings are right to set up the race. > Simple fix right now is make sure the callback captures a shared_ptr to the > promise. Longer term fix may be to make a class that encapsulates the > promise and future with a reference counting mechanism to prevent more of > these bugs and use that instead of the std lib versions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11730) libhdfs++: RPC connection should handle authorization error call id
[ https://issues.apache.org/jira/browse/HDFS-11730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410462#comment-16410462 ] Hudson commented on HDFS-11730: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11730: libhdfs++: RpcConnection should handle authorization error (james.clampffer: rev b584e34f2f626c72b1a632bfc82da1247ff438d8) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc > libhdfs++: RPC connection should handle authorization error call id > --- > > Key: HDFS-11730 > URL: https://issues.apache.org/jira/browse/HDFS-11730 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Minor > Attachments: HDFS-11730.HDFS-8707.000.patch > > > 2 Problems: > 1) Authorization error call id isn't recognized, so retry logic kicks in > rather than bailing out when the client gets it > 2) Printing call_id as unsigned during error means small negative values used > for signalling look like large positive values. This patch also just casts > that error back to a signed 32 bit int. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11027) libhdfs++: Don't retry if there is an authentication failure
[ https://issues.apache.org/jira/browse/HDFS-11027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410452#comment-16410452 ] Hudson commented on HDFS-11027: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11027: libbhdfs++: Don't retry if there is an authentication (james.clampffer: rev b7d9a0f827c0688fa1ebe6b71f0e0a4839d73ea2) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc > libhdfs++: Don't retry if there is an authentication failure > > > Key: HDFS-11027 > URL: https://issues.apache.org/jira/browse/HDFS-11027 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11027.HDFS-8707.000.patch, > HDFS-11027.HDFS-8707.001.patch > > > "Authentication failed" status falls into the general !status.ok() block in > the HA retry policy so it will keep attempting to failover. If the client > isn't kerberized, or doesn't have the right ticket it should give up and > return a meaningful error message (right now it returns a generic bad > connection failure string). > Wouldn't hurt to check the FixedDelayRetryPolicy to make sure that doesn't > also keep attempting to retry in the same way. I suspect it does. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10685) libhdfs++: return explicit error when non-secured client connects to secured server
[ https://issues.apache.org/jira/browse/HDFS-10685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410457#comment-16410457 ] Hudson commented on HDFS-10685: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10685: libhdfs++: return explicit error when non-secured client (james.clampffer: rev 58de2df8606cd598b41785c0c5c6bf98d66ebc15) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc > libhdfs++: return explicit error when non-secured client connects to secured > server > --- > > Key: HDFS-10685 > URL: https://issues.apache.org/jira/browse/HDFS-10685 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Kai Jiang >Priority: Major > Attachments: HDFS-10685.HDFS-8707.000.patch, > HDFS-10685.HDFS-8707.001.patch > > > When a non-secured client tries to connect to a secured server, the first > indication is an error from RpcConnection::HandleRpcRespose complaining about > "RPC response with Unknown call id -33". > We should insert code in HandleRpcResponse to detect if the unknown call id > == RpcEngine::kCallIdSasl and return an informative error that you have an > unsecured client connecting to a secured server. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9271) Implement basic NN operations
[ https://issues.apache.org/jira/browse/HDFS-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410441#comment-16410441 ] Hudson commented on HDFS-9271: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-9271: libhdfs++: Implement basic NN operations. Contributed by (james.clampffer: rev 2a8edd4e523a2e2e90ee570d0c61a600cf7bf610) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/options.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.h > Implement basic NN operations > - > > Key: HDFS-9271 > URL: https://issues.apache.org/jira/browse/HDFS-9271 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-9271.HDFS-8707.000.patch, > HDFS-9271.HDFS-8707.001.patch, HDFS-9271.HDFS-8707.002.patch, > HDFS-9271.HDFS-8707.003.patch, HDFS-9271.HDFS-8707.004.patch, > HDFS-9271.HDFS-8707.005.patch, HDFS-9271.HDFS-8707.006.patch, > HDFS-9271.HDFS-8707.007.patch, HDFS-9271.HDFS-8707.008.patch > > > Expose via C and C++ API: > * mkdirs > * rename > * delete > * stat > * chmod > * chown > * getListing > * setOwner -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10450) libhdfs++: Implement Cyrus SASL implementation in sasl_enigne.cc
[ https://issues.apache.org/jira/browse/HDFS-10450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410447#comment-16410447 ] Hudson commented on HDFS-10450: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10450: libhdfs++: Add Cyrus SASL support. Contributed by Don (james.clampffer: rev 549a5dbce9f6aa398908e29e05c0a7e85f3cdc23) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc * (edit) dev-support/docker/Dockerfile * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.h HDFS-10450b: libhdfs++: Add Cyrus SASL support (engines). Contributed (james.clampffer: rev d22e4b2eb77673725eece896bc52ccdce692d987) * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/gsasl_engine.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/gsasl_engine.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/cyrus_sasl_engine.cc > libhdfs++: Implement Cyrus SASL implementation in sasl_enigne.cc > > > Key: HDFS-10450 > URL: https://issues.apache.org/jira/browse/HDFS-10450 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-10450.HDFS-8707.000.patch, > HDFS-10450.HDFS-8707.001.patch, HDFS-10450.HDFS-8707.002.patch, > HDFS-10450.HDFS-8707.003.patch, HDFS-10450.HDFS-8707.004.patch, > HDFS-10450.HDFS-8707.004.patch, HDFS-10450.HDFS-8707.005.patch > > > The current sasl_engine implementation was proven out using GSASL, which is > does not have an ASF-approved license. It included a framework to use Cyrus > SASL (libsasl2.so) instead; we should complete that implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12026) libhdfs++: Fix compilation errors and warnings when compiling with Clang
[ https://issues.apache.org/jira/browse/HDFS-12026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410473#comment-16410473 ] Hudson commented on HDFS-12026: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12026: libhdfs++: Fix compilation errors and warnings when (james.clampffer: rev 033433bce76101a626b3376404aaf85f62a82711) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/configuration.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/block_location.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/optional_wrapper.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/node_exclusion_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ioservice_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/bad_datanode_tracker.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_engine.h * (edit) dev-support/docker/Dockerfile * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/auth_info.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/libhdfs_events_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/pom.xml > libhdfs++: Fix compilation errors and warnings when compiling with Clang > - > > Key: HDFS-12026 > URL: https://issues.apache.org/jira/browse/HDFS-12026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-12026.HDFS-8707.000.patch, > HDFS-12026.HDFS-8707.001.patch, HDFS-12026.HDFS-8707.002.patch, > HDFS-12026.HDFS-8707.003.patch, HDFS-12026.HDFS-8707.004.patch, > HDFS-12026.HDFS-8707.005.patch, HDFS-12026.HDFS-8707.006.patch, > HDFS-12026.HDFS-8707.007.patch, HDFS-12026.HDFS-8707.008.patch, > HDFS-12026.HDFS-8707.009.patch, HDFS-12026.HDFS-8707.010.patch > > > Currently multiple errors and warnings prevent libhdfspp from being compiled > with clang. It should compile cleanly using flag: > -std=c++11 > and also warning flags: > -Weverything -Wno-c++98-compat -Wno-missing-prototypes > -Wno-c++98-compat-pedantic -Wno-padded -Wno-covered-switch-default > -Wno-missing-noreturn -Wno-unknown-pragmas -Wconversion -Werror -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10746) libhdfs++: synchronize access to working_directory and bytes_read_.
[ https://issues.apache.org/jira/browse/HDFS-10746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410443#comment-16410443 ] Hudson commented on HDFS-10746: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10746: libhdfs++: synchronize access to working_directory and (james.clampffer: rev b9cf0e932d8a56eadc9c23912590722a5d846dec) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc > libhdfs++: synchronize access to working_directory and bytes_read_. > --- > > Key: HDFS-10746 > URL: https://issues.apache.org/jira/browse/HDFS-10746 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-10746.HDFS-8707.000.patch, > HDFS-10746.HDFS-8707.001.patch > > > std::string working_directory is located in hdfs.cc and access to it should > be synchronized with locks. > uint64_t bytes_read_; is located in filehandle.h and it should be made atomic > in order to be thread safe when multithreading becomes available. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10785) libhdfs++: Implement the rest of the tools
[ https://issues.apache.org/jira/browse/HDFS-10785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410465#comment-16410465 ] Hudson commented on HDFS-10785: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10785: libhdfs++: Implement the rest of the tools. Contributed by (james.clampffer: rev 869317be0a1fdff23be5fc500dcd9ae4ecd7bc29) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_get.cc * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/CMakeLists.txt * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/gendirs/gendirs.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_createSnapshot.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_count.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_stat.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chown.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_disallowSnapshot.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_moveToLocal.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_df.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/find/find.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_rm.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/cat/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_cat.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/CMakeLists.txt * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chgrp.cc * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/cat/cat.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/c/cat/cat.c * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_mkdir.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_du.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/fsinfo.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/statinfo.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/gendirs/gendirs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/tools_common.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_find.cpp * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/libhdfspp_wrapper_defines.h * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tools/hdfs_chmod.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/statinfo.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem_sync.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/gendirs/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/find/find.cpp * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cc/connect_cancel/connect_cancel.cc * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/cat/CMakeLists.txt * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cpp/connect_cancel/con
[jira] [Commented] (HDFS-10526) libhdfs++: Add connect timeouts to async_connect calls
[ https://issues.apache.org/jira/browse/HDFS-10526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410432#comment-16410432 ] Hudson commented on HDFS-10526: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10526: libhdfs++: Add connect timeouts to async_connect calls. (james.clampffer: rev 2616fe2025af642b25e61ff9f9a9a2624a8bbd24) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_configuration_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/options.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h > libhdfs++: Add connect timeouts to async_connect calls > -- > > Key: HDFS-10526 > URL: https://issues.apache.org/jira/browse/HDFS-10526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-10526.HDFS-8707.000.patch, > HDFS-10526.HDFS-8707.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10311) libhdfs++: DatanodeConnection::Cancel should not delete the underlying socket
[ https://issues.apache.org/jira/browse/HDFS-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410421#comment-16410421 ] Hudson commented on HDFS-10311: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10311: DatanodeConnection::Cancel should not delete the underlying (james.clampffer: rev e542db66ee4465744e019a92ba410f3e20d09365) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.cc > libhdfs++: DatanodeConnection::Cancel should not delete the underlying socket > - > > Key: HDFS-10311 > URL: https://issues.apache.org/jira/browse/HDFS-10311 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10311.HDFS-8707.000.patch, > HDFS-10311.HDFS-8707.001.patch, HDFS-10311.HDFS-8707.002.patch > > > DataNodeConnectionImpl calls reset on the unique_ptr that references the > underlying asio::tcp::socket. If this happens after the continuation > pipeline checks the cancel state but before asio uses the socket it will > segfault because unique_ptr::reset will explicitly change it's value to > nullptr. > Cancel should only call shutdown() and close() on the socket but keep the > instance of it alive. The socket can probably also be turned into a member > of DataNodeConnectionImpl to get rid of the unique pointer and simplify > things a bit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10931) libhdfs++: Fix object lifecycle issues in the BlockReader
[ https://issues.apache.org/jira/browse/HDFS-10931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410449#comment-16410449 ] Hudson commented on HDFS-10931: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10931: libhdfs++: Fix object lifecycle issues in the BlockReader (james.clampffer: rev 2a42eeb66f7fbf3fb4fc434480a712c53cf0243a) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_mini_stress.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc > libhdfs++: Fix object lifecycle issues in the BlockReader > - > > Key: HDFS-10931 > URL: https://issues.apache.org/jira/browse/HDFS-10931 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Critical > Attachments: HDFS-10931.HDFS-8707.000.patch, > HDFS-10931.HDFS-8707.001.patch > > > The BlockReader can work itself into a a state during AckRead (possibly other > stages as well) where the pipeline posts a task for asio with a pointer back > into itself, then promptly calls "delete this" without canceling the asio > request. The asio task finishes and tries to acquire the lock at the address > where the DataNodeConnection used to live - but the DN connection is no > longer valid so it's scribbling on some arbitrary bit of memory. On some > platforms the underlying address used by the mutex state will be handed out > to future mutexes so the scribble breaks that state and all the locks in that > process start misbehaving. > This can be reproduced by using the patch from HDFS-8790 and adding more > worker threads + a lot more reader threads. > I'm going to fix this in two parts: > 1) Duct tape + superglue patch to make sure that all top level continuations > in the block reader pipeline hold a shared_ptr to the DataNodeConnection. > Nested continuations also get a copy of the shared_ptr to make sure the > connection is alive. This at least keeps the connection alive so that it can > keep returning asio::operation_aborted. > 2) The continuation stuff needs a lot of work to make sure this type of bug > doesn't keep popping up. We've already fixed these issues in the RPC code. > This will most likely need to be split into a few jiras. > - Continuation "framework" can be slimmed down quite a bit, perhaps even > removed. Near zero documentation + many implied contracts = constant bug > chasing. > - Add comments to actually describe what's going on in the networking code. > This bug took significantly longer than it should have to track down because > I hadn't worked on the BlockReader in a while. > - No more "delete this". > - Flatten out nested continuations e.g. the guts of BlockReaderImpl::AckRead. > It's unclear why they were implemented like this in the first place and > there's no comments to indicate that this was intentional. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10247) libhdfs++: Datanode protocol version mismatch
[
https://issues.apache.org/jira/browse/HDFS-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410420#comment-16410420
]
Hudson commented on HDFS-10247:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13869/])
HDFS-10247: libhdfs++: Datanode protocol version mismatch fix.
(james.clampffer: rev 60c3437267b864a01d27783535906c6a7e81058e)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/asio.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/continuation/protobuf.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.h
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/datatransfer_impl.h
> libhdfs++: Datanode protocol version mismatch
> -
>
> Key: HDFS-10247
> URL: https://issues.apache.org/jira/browse/HDFS-10247
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: James Clampffer
>Assignee: James Clampffer
>Priority: Major
> Attachments: HDFS-10247.HDFS-8707.000.patch,
> HDFS-10247.HDFS-8707.001.patch, HDFS-10247.HDFS-8707.002.patch
>
>
> Occasionally "Version Mismatch (Expected: 28, Received: 22794 )" shows up in
> the logs. This doesn't happen much at all with less than 500 concurrent
> reads and starts happening often enough to be an issue at 1000 concurrent
> reads.
> I've seen 3 distinct numbers: 23050 (most common), 22538, and 22794. If you
> break these shorts into bytes you get
> {code}
> 23050 -> [90,10]
> 22794 -> [89,10]
> 22538 -> [88,10]
> {code}
> Interestingly enough if we dump buffers holding protobuf messages just before
> they hit the wire we see things like the following with the first two bytes
> as 90,10
> {code}
> buffer
> ={90,10,82,10,64,10,52,10,37,66,80,45,49,51,56,49,48,51,51,57,57,49,45,49,50,55,46,48,46,48,46,49,45,49,52,53,57,53,50,53,54,49,53,55,50,53,16,-127,-128,-128,-128,4,24,-23,7,32,-128,-128,64,18,8,10,0,18,0,26,0,34,0,18,14,108,105,98,104,100,102,115,43,43,95,75,67,43,49,16,0,24,23,32,1}
> {code}
> The first 3 bytes the DN is expecting for an unsecured read block request =
> {code}
> {0,28,81} //[0, 28]->a short for protocol, 81 is read block opcode
> {code}
> This seems like either connections are getting swapped between readers or
> the header isn't being sent for some reason but the protobuf message is.
> I've ruled out memory stomps on the header data (see HDFS-10241) by sticking
> the 3 byte header in it's own static buffer that all requests use.
> Some notes:
> -The mismatched number will stay the same for the duration of a stress test.
> -The mismatch is distributed fairly evenly throughout the logs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10705) libhdfs++: FileSystem should have a convenience no-args ctor
[ https://issues.apache.org/jira/browse/HDFS-10705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410446#comment-16410446 ] Hudson commented on HDFS-10705: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10705: libhdfs++: FileSystem should have a convenience no-args (james.clampffer: rev 9d1c9021591484145d14e889093fb773cc4b0557) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h > libhdfs++: FileSystem should have a convenience no-args ctor > > > Key: HDFS-10705 > URL: https://issues.apache.org/jira/browse/HDFS-10705 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10705.HDFS-8707.000.patch, > HDFS-10705.HDFS-8707.001.patch > > > Our examples demonstrate that the common use case is "use default options, > default username, and default IOService." Let's make a FileSystem::New() > that helps users with that. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10511) libhdfs++: make error returning mechanism consistent across all hdfs operations
[ https://issues.apache.org/jira/browse/HDFS-10511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410433#comment-16410433 ] Hudson commented on HDFS-10511: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10511: libhdfs++: make error returning mechanism consistent across (james.clampffer: rev 193314dc34c44b4ba34a5f83b9346b2d1e96c5d2) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc > libhdfs++: make error returning mechanism consistent across all hdfs > operations > --- > > Key: HDFS-10511 > URL: https://issues.apache.org/jira/browse/HDFS-10511 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-10511.HDFS-8707.000.patch, > HDFS-10511.HDFS-8707.000.patch, HDFS-10511.HDFS-8707.001.patch, > HDFS-10511.HDFS-8707.002.patch, HDFS-10511.HDFS-8707.003.patch > > > Errno should always be set. > If function is returning a code on stack, it should be consistent with errno. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11014) libhdfs++: Make connection to HA clusters faster
[ https://issues.apache.org/jira/browse/HDFS-11014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410451#comment-16410451 ] Hudson commented on HDFS-11014: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11014: libhdfs++: Make connection to HA clusters faster. (james.clampffer: rev 59a39269463bc6fd76b1e5e30cc8ccde5250e7fb) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h > libhdfs++: Make connection to HA clusters faster > > > Key: HDFS-11014 > URL: https://issues.apache.org/jira/browse/HDFS-11014 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Minor > Attachments: HDFS-11014.HDFS-8707.000.patch > > > Right now when we get a StandbyException from the NN we inject a 20 second > delay before we try the alternate NN even if it's the first failover. The > first failover shouldn't have a delay (java client skips delay on first > failover). > Another minor change I'd like to make is to reduce the default number of > failover attempts from 15 (used in the apache config) to 4. My impression is > that higher numbers of failovers are really handy for longer running batch > jobs but in the libhdfs++ case the client is often an interactive > application. In this case it's generally preferable to fail sooner so a user > doesn't have to wait the ~8 minutes to time out when using default settings. > 4 failovers is based on the assumption that if we can't immediately connect > there is either a GC pause which will most likely be finished before the > second connection attempt or it's a network or config issue that will take > some sorting out by an admin. It'd still be possible to override these in > the config for more tuning if a specific deployment tends to have more or > less network issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10464) libhdfs++: Implement GetPathInfo and ListDirectory
[ https://issues.apache.org/jira/browse/HDFS-10464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410427#comment-16410427 ] Hudson commented on HDFS-10464: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10464: libhdfs++: Implement GetPathInfo and ListDirectory. (james.clampffer: rev 0496bc54641d3ce284ab8832099c6a9d67ee9069) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/statinfo.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc > libhdfs++: Implement GetPathInfo and ListDirectory > -- > > Key: HDFS-10464 > URL: https://issues.apache.org/jira/browse/HDFS-10464 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-10464.HDFS-8707.000.patch, > HDFS-10464.HDFS-8707.001.patch, HDFS-10464.HDFS-8707.002.patch, > HDFS-10464.HDFS-8707.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10454) libhdfspp: Move NameNodeOp to a separate file
[ https://issues.apache.org/jira/browse/HDFS-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410428#comment-16410428 ] Hudson commented on HDFS-10454: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10454: libhdfspp: Move NameNodeOp to a separate file. Contributed (james.clampffer: rev 50aa4ea7760bf2b0c59c467861c7ed2bdc1ea97e) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h > libhdfspp: Move NameNodeOp to a separate file > - > > Key: HDFS-10454 > URL: https://issues.apache.org/jira/browse/HDFS-10454 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Minor > Attachments: HDFS-10454.HDFS-8707.000.patch, > HDFS-10454.HDFS-8707.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12013) libhdfs++: read with offset at EOF should return 0 bytes instead of error
[ https://issues.apache.org/jira/browse/HDFS-12013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410468#comment-16410468 ] Hudson commented on HDFS-12013: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12013: libhdfs++: read with offset at EOF should return 0 bytes (james.clampffer: rev 3c743b475e6ec431a5849ebd467b077e41e89987) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc > libhdfs++: read with offset at EOF should return 0 bytes instead of error > - > > Key: HDFS-12013 > URL: https://issues.apache.org/jira/browse/HDFS-12013 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Xiaowei Zhu >Assignee: Xiaowei Zhu >Priority: Critical > Attachments: HDFS-12013.HDFS-8707.000.patch > > > The current behavior is when you read from offset == file_length, it will > throw error: > "AsyncPreadSome: trying to begin a read past the EOF" > But read with offset at EOF should just return 0 bytes. The above error > should only be thrown when offset > file_length. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9118) Add logging system for libdhfs++
[ https://issues.apache.org/jira/browse/HDFS-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410416#comment-16410416 ] Hudson commented on HDFS-9118: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-9118: libhdfs++ Add logging system. Contributed by James Clampffer (james.clampffer: rev 0f1a278dd5f2d3cafee240f2c2795ac0644410a2) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/log.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/logging_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_public_api.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfs_ext.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h > Add logging system for libdhfs++ > > > Key: HDFS-9118 > URL: https://issues.apache.org/jira/browse/HDFS-9118 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Affects Versions: HDFS-8707 >Reporter: Bob Hansen >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-9118.HDFS-8707.000.patch, > HDFS-9118.HDFS-8707.001.patch, HDFS-9118.HDFS-8707.002.patch, > HDFS-9118.HDFS-8707.003.patch, HDFS-9118.HDFS-8707.003.patch, > HDFS-9118.HDFS-8707.004.patch > > > With HDFS-9505, we've starting logging data from libhdfs++. Consumers of the > library are going to have their own logging infrastructure that we're going > to want to provide data to. > libhdfs++ should have a logging library that: > * Is overridable and can provide sufficient information to work well with > common C++ logging frameworks > * Has a rational default implementation > * Is performant -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11305) libhdfs++: Log Datanode information when reading an HDFS block
[ https://issues.apache.org/jira/browse/HDFS-11305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410456#comment-16410456 ] Hudson commented on HDFS-11305: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11305: libhdfs++: Log Datanode information when reading an HDFS (james.clampffer: rev 69d55340f89a9bf2efc16b00b24c1ebf303e89fe) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc > libhdfs++: Log Datanode information when reading an HDFS block > -- > > Key: HDFS-11305 > URL: https://issues.apache.org/jira/browse/HDFS-11305 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Xiaowei Zhu >Assignee: Xiaowei Zhu >Priority: Minor > Attachments: HDFS-11305.HDFS-8707.000.patch, > HDFS-11305.HDFS-8707.001.patch > > > The information can be logged as debug log and contain hostname, ip address, > with file path and offset. With these information, we can check things like > rack locality, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11908) libhdfs++: Authentication failure when first NN of kerberized HA cluster is standby
[ https://issues.apache.org/jira/browse/HDFS-11908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410470#comment-16410470 ] Hudson commented on HDFS-11908: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11908: libhdfs++: Authentication failure when first NN of a (james.clampffer: rev 23afd508651c5e4a10cf2c4784b543a6e77f3791) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h > libhdfs++: Authentication failure when first NN of kerberized HA cluster is > standby > --- > > Key: HDFS-11908 > URL: https://issues.apache.org/jira/browse/HDFS-11908 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11908.HDFS-8707.000.patch > > > Library won't properly authenticate to kerberized HA cluster if the first > namenode it tries to connect to is the standby. RpcConnection ends up > attempting to use simple auth. > Control flow to connect to NN for the first time: > # RpcConnection constructed with a pointer to the RpcEngine as the only > argument > # RpcConnection::Connect(server endpoints, auth_info, callback called) > ** auth_info contains the SASL mechanism to use + the delegation token if we > already have one > Control flow to connect to NN after failover: > # RpcEngine::NewConnection called, allocates an RpcConnection exactly how > step 1 above would > # RpcEngine::InitializeConnection called, sets event hooks and a string for > cluster name > # Rpc calls sent using RpcConnection::PreEnqueueRequests called to add RPC > message that didn't make it on last call due to standby exception > # RpcConnection::ConnectAndFlush called to send RPC packets. This only takes > server endpoints, no auth info > To fix: > RpcEngine::InitializeConnection just needs to set RpcConnection::auth_info_ > from the existing RpcEngine::auth_info_, even better would be setting this in > the constructor so if an RpcConnection exists it can be expected to be in a > usable state. I'll get a diff up once I sort out CI build failures. > Also really need to get CI test coverage for HA and kerberos because this > issue should not have been around for so long. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10898) libhdfs++: Make log levels consistent
[ https://issues.apache.org/jira/browse/HDFS-10898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410450#comment-16410450 ] Hudson commented on HDFS-10898: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10898: libhdfs++: Make log levels consistent. Contributed by James (james.clampffer: rev 7ebecaeeded9ef6f0d9dff3e8994cf6eaad18920) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc > libhdfs++: Make log levels consistent > - > > Key: HDFS-10898 > URL: https://issues.apache.org/jira/browse/HDFS-10898 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Trivial > Attachments: HDFS-10898.HDFS-8707.000.patch, > HDFS-10898.HDFS-8707.001.patch > > > Most of the public C++ FileHandle/FileSystem operations have a LOG_TRACE > level message about parameters passed in etc. However many methods use > LOG_DEBUG and a couple use LOG_INFO. > We most likely want FS operations that happen a lot (read/open/seek/stat) to > stick to LOG_DEBUG consistently and only use LOG_INFO for things like > FileSystem::Connect or RpcConnection:: that don't get called often and are > important enough to warrant showing up in the log. LOG_TRACE can be reserved > for things happening deeper inside public methods and methods that aren't > part of the public API. > Related improvements that could be brought into this to avoid opening a ton > of small Jiras: > -Print the "this" pointer address in the log message to make it easier to > correlate objects when there's concurrent work being done. This has been > very helpful in the past but often got stripped out before patches went in. > People just need be aware that operator new may eventually place an object of > the same type at the same address sometime in the future. > -For objects owned by other objects, but created on the fly, include a > pointer back to the parent/creator object if that pointer is already being > tracked (See the nested stucts in BlockReaderImpl). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11449) libhdfs++: Log Datanode read size when reading an HDFS block
[ https://issues.apache.org/jira/browse/HDFS-11449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410460#comment-16410460 ] Hudson commented on HDFS-11449: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11449: libhdfs++: Log Datanode read size when reading from an HDFS (james.clampffer: rev 0f3f8db113621e177dbb7fc3e93f4870bff2d739) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc > libhdfs++: Log Datanode read size when reading an HDFS block > > > Key: HDFS-11449 > URL: https://issues.apache.org/jira/browse/HDFS-11449 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Xiaowei Zhu >Assignee: Xiaowei Zhu >Priority: Trivial > Attachments: HDFS-11449.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10332) hdfs-native-client fails to build with CMake 2.8.11 or earlier
[
https://issues.apache.org/jira/browse/HDFS-10332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410423#comment-16410423
]
Hudson commented on HDFS-10332:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13869/])
HDFS-10332. hdfs-native-client fails to build with CMake 2.8.11 or
(james.clampffer: rev b9eeecc477ead34b8c350043bbf62386b0af18e4)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt
> hdfs-native-client fails to build with CMake 2.8.11 or earlier
> --
>
> Key: HDFS-10332
> URL: https://issues.apache.org/jira/browse/HDFS-10332
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs-client
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: Minor
> Attachments: HDFS-10332.01.patch, HDFS-10332.HDFS-8707.001.patch
>
>
> Due to a new syntax introduced in CMake 2.8.12 (get_filename_component's
> function when VAR=DIRECTORY) the native-client won't build.
> Currently RHEL6 & 7 are using older version of CMake.
> Error log:
> {noformat}
> [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-hdfs-native-client ---
> [INFO] Executing tasks
> main:
> [exec] JAVA_HOME=,
> JAVA_JVM_LIBRARY=/usr/java/jdk1.7.0_79/jre/lib/amd64/server/libjvm.so
> [exec] JAVA_INCLUDE_PATH=/usr/java/jdk1.7.0_79/include,
> JAVA_INCLUDE_PATH2=/usr/java/jdk1.7.0_79/include/linux
> [exec] Located all JNI components successfully.
> [exec] -- Could NOT find PROTOBUF (missing: PROTOBUF_LIBRARY
> PROTOBUF_INCLUDE_DIR)
> [exec] -- valgrind location: MEMORYCHECK_COMMAND-NOTFOUND
> [exec] -- checking for module 'fuse'
> [exec] -- package 'fuse' not found
> [exec] -- Failed to find Linux FUSE libraries or include files. Will
> not build FUSE client.
> [exec] -- Configuring incomplete, errors occurred!
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:95 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:96 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:97 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error at main/native/libhdfspp/CMakeLists.txt:71
> (get_filename_component):
> [exec] get_filename_component unknown component DIRECTORY
> [exec] Call Stack (most recent call first):
> [exec] main/native/libhdfspp/CMakeLists.txt:98 (copy_on_demand)
> [exec]
> [exec]
> [exec] CMake Error: The following variables are used in this project,
> but they are set to NOTFOUND.
> [exec] Please set them or make sure they are set and tested correctly in
> the CMake files:
> [exec] PROTOBUF_LIBRARY (ADVANCED)
> [exec] linked by target "hdfspp" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp
> [exec] linked by target "hdfspp_static" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp
> [exec] linked by target "protoc-gen-hrpc" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/proto
> [exec] linked by target "bad_datanode_test" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "hdfs_builder_test" in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "hdfspp_errors_test" in directory
> /home/tiborkiss/devel/workspace
> [exec]
> /hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests
> [exec] linked by target "libhdfs_threaded_hdfspp_test_shim_static"
> in directory
> /home/tiborkiss/devel/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/test
> [e
[jira] [Commented] (HDFS-10524) libhdfs++: Implement chmod and chown
[ https://issues.apache.org/jira/browse/HDFS-10524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410435#comment-16410435 ] Hudson commented on HDFS-10524: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10524: libhdfs++: Implement chmod and chown. Contributed by (james.clampffer: rev d4c3cfbf47ccc6d033cf9aa16b2eb16809b886ef) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/hdfspp.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc > libhdfs++: Implement chmod and chown > > > Key: HDFS-10524 > URL: https://issues.apache.org/jira/browse/HDFS-10524 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Anatoli Shein >Priority: Major > Attachments: HDFS-10524.HDFS-8707.000.patch, > HDFS-10524.HDFS-8707.001.patch, HDFS-10524.HDFS-8707.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11106) libhdfs++: Some refactoring to better organize files
[ https://issues.apache.org/jira/browse/HDFS-11106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410454#comment-16410454 ] Hudson commented on HDFS-11106: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11106: libhdfs++: Some refactoring to better organize files. (james.clampffer: rev dd7837c429f689e45f91248c1ee5b7d10e168fbf) * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_public_api.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/base64.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem_sync.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/sasl_digest_md5_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_public_api.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_ioservice.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_ioservice.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/CMakeLists.txt HDFS-11106: libhdfs++: Some refactoring to better organize files (part (james.clampffer: rev 3e53da2d623cfecc9f6af0fcf39413edb46b4092) * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/namenode_tracker.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/request.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (delete) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/request.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/namenode_tracker.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc > libhdfs++: Some refactoring to better organize files > > > Key: HDFS-11106 > URL: https://issues.apache.org/jira/browse/HDFS-11106 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11106.HDFS-8707.000.patch, > HDFS-11106.HDFS-8707.001.patch > > > I propose splitting some of the files that have grown wild over time into > files that align with more specific functionality. It's probably best to do > this in a few pieces so it doesn't invalidate anyone's patches in progress. > Here's what I have in mind, looking for feedback if 1) it's not worth doing > for some reason 2) it will break your patch and you'd like this to wait. I'd > also like to consolidate related functions, mostly protobuf helpers, that are > spread around the library into dedicated files. > Targets (can split each into a separate patch): > * (done in patch 000, committed) separate the implementation of operations > from async shim code in files like filesystem.cc (make a > filesystem_shims.cc). The shims are just boilerplate code that only need to > change if the signature of their async counterparts change. > * (done in patch 000, committed) merge base64.cc into util.cc; base64.cc only > conta
[jira] [Commented] (HDFS-10231) libhdfs++: Fix race conditions in RPC layer
[ https://issues.apache.org/jira/browse/HDFS-10231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410419#comment-16410419 ] Hudson commented on HDFS-10231: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10231: libhdfs++: Fix race conditions in RPC layer. Contributed by (james.clampffer: rev ed77d8d5df23be776e87032c9667b9025e889504) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc > libhdfs++: Fix race conditions in RPC layer > --- > > Key: HDFS-10231 > URL: https://issues.apache.org/jira/browse/HDFS-10231 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-10231.HDFS-8707.000.patch, > HDFS-10231.HDFS-8707.001.patch, HDFS-10231.HDFS-8707.002.patch, > HDFS-10231.HDFS-8707.003.patch > > > Namenode calls seem to hang and the retry logic never properly kicks in. It > looks like there's a race condition that leads to a failed rpc call never > properly passing the request object to the new RpcConnectionImpl so things > hang forever. RpcConnectionImpl objects are also kept alive longer than they > should because of a shared_ptr cycle between them and the timeout tracking > objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12103) libhdfs++: Provide workaround to support cancel on filesystem connect until HDFS-11437 is resolved
[ https://issues.apache.org/jira/browse/HDFS-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410471#comment-16410471 ] Hudson commented on HDFS-12103: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-12103: libhdfs++: Provide workaround to support cancel on (james.clampffer: rev 3db261d45bab46759f4049675483633822b1d090) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc > libhdfs++: Provide workaround to support cancel on filesystem connect until > HDFS-11437 is resolved > -- > > Key: HDFS-12103 > URL: https://issues.apache.org/jira/browse/HDFS-12103 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-12103.HDFS-8707.000.patch > > > HDFS-11437 is going to take a non-trivial amount of work to do right. In the > meantime it'd be nice to have a way to cancel pending connections (even when > the FS claimed they are finished). > Proposed workaround is to relax the rules about when > FileSystem::CancelPending connect can be called since it isn't able to > properly determine when it's connected anyway. In order to determine when > the FS has connected you can do some simple RPC call since that will wait on > failover. If CancelPending can be called during that first RPC call then it > will effectively be canceling FileSystem::Connect > Current cancel rules - asterisk on steps where CancelPending is allowed > FileSystem::Connect called > FileSystem communicates with first NN * > FileSystem::Connect returns - even if it hasn't communicated with the active > NN > Proposed relaxation > FileSystem::Connect called > FileSystem communicates with first NN* > FileSystem::Connect returns * > FileSystem::GetFileInfo called * -any namenode RPC call will do, ignore perm > errors > RPC engine blocks until it hits the active or runs out of retries * > FileSystem::GetFileInfo returns > It'd be up to the user to add in the dummy NN RPC call. Once HDFS-11437 is > fixed this workaround can be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10441) libhdfs++: HA namenode support
[ https://issues.apache.org/jira/browse/HDFS-10441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410438#comment-16410438 ] Hudson commented on HDFS-10441: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10441: libhdfs++: HA namenode support. Contributed by James (james.clampffer: rev 4cb0dad5e535f4ee79c32a54f8958bf7d4402123) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/uri.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/CMakeLists.txt * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/status.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/namenode_operations.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filesystem.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.h * (add) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/namenode_info.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/logging.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/status.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/uri.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/options.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/options.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/events.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/hdfs_configuration.h > libhdfs++: HA namenode support > -- > > Key: HDFS-10441 > URL: https://issues.apache.org/jira/browse/HDFS-10441 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10441.HDFS-8707.000.patch, > HDFS-10441.HDFS-8707.002.patch, HDFS-10441.HDFS-8707.003.patch, > HDFS-10441.HDFS-8707.004.patch, HDFS-10441.HDFS-8707.005.patch, > HDFS-10441.HDFS-8707.006.patch, HDFS-10441.HDFS-8707.007.patch, > HDFS-10441.HDFS-8707.008.patch, HDFS-10441.HDFS-8707.009.patch, > HDFS-10441.HDFS-8707.010.patch, HDFS-10441.HDFS-8707.011.patch, > HDFS-10441.HDFS-8707.012.patch, HDFS-10441.HDFS-8707.013.patch, > HDFS-10441.HDFS-8707.014.patch, HDFS-8707.HDFS-10441.001.patch > > > If a cluster is HA enabled then do proper failover. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10222) libhdfs++: Shutdown sockets to avoid "Connection reset by peer"
[ https://issues.apache.org/jira/browse/HDFS-10222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410418#comment-16410418 ] Hudson commented on HDFS-10222: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10222. libhdfs++: Shutdown sockets to avoid 'Connection reset by (james.clampffer: rev 015d93a71527b41c5de22ed6b0efa04636ed4462) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.h > libhdfs++: Shutdown sockets to avoid "Connection reset by peer" > --- > > Key: HDFS-10222 > URL: https://issues.apache.org/jira/browse/HDFS-10222 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10222.HDFS-8707.000.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10595) libhdfs++: Client Name Protobuf Error
[ https://issues.apache.org/jira/browse/HDFS-10595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410448#comment-16410448 ] Hudson commented on HDFS-10595: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10595: libhdfs++: deconflate client name and client id (james.clampffer: rev 69cb05d2920a02435921028b70ed88c587823c05) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc > libhdfs++: Client Name Protobuf Error > - > > Key: HDFS-10595 > URL: https://issues.apache.org/jira/browse/HDFS-10595 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Anatoli Shein >Assignee: Bob Hansen >Priority: Major > Attachments: HDFS-10595.HDFS-8707.patch.000 > > > When running a cat tool > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/examples/cat/c/cat.c) I > get the following error: > [libprotobuf ERROR google/protobuf/wire_format.cc:1053] String field contains > invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type > if you intend to send raw bytes. > However it executes correctly. Looks like this error happens when trying to > serialize Client name in ClientOperationHeaderProto::SerializeWithCachedSizes > (/hadoop-hdfs-native-client/target/main/native/libhdfspp/lib/proto/datatransfer.pb.cc) > Possibly the problem is caused by generating client name as a UUID in > GetRandomClientName > (/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/util.cc) > In Java client it looks like there are two different unique client > identifiers: ClientName and ClientId: > Client name is generated as: > clientName = "DFSClient_" + dfsClientConf.getTaskId() + "_" + > ThreadLocalRandom.current().nextInt() + "_" + > Thread.currentThread().getId(); > (/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java) > ClientId is generated as a UUID in > (/hadoop-common/src/main/java/org/apache/hadoop/ipc/ClientId.java) > In libhdfs++ we need to possibly also have two unique client identifiers, or > fix the current client name to work without protobuf warnings/errors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10761) libhdfs++: Fix broken logic in HA retry policy
[ https://issues.apache.org/jira/browse/HDFS-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410444#comment-16410444 ] Hudson commented on HDFS-10761: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-10761: libhdfs++: Fix broken logic in HA retry policy. Contributed (james.clampffer: rev 05ddb3108176a32cfda945ba4cb736c91b38877b) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/retry_policy.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_engine.cc > libhdfs++: Fix broken logic in HA retry policy > -- > > Key: HDFS-10761 > URL: https://issues.apache.org/jira/browse/HDFS-10761 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-10761.HDFS-8707.000.patch > > > Two issues in the HA policy: > 1) There's logic to guard against HDFS-8161, but it wasn't checking how many > failovers had already happened. It'd get stuck alternating between nodes > forever. > 2) Switched a ">" for a "<" so when max failovers was reached it'd keep > trying anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11767) libhdfs++: SASL events should be scoped closer to usage
[ https://issues.apache.org/jira/browse/HDFS-11767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410463#comment-16410463 ] Hudson commented on HDFS-11767: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11767: libhdfs++: SASL events should be scoped closer to usage. (james.clampffer: rev 57cdad73deecdb39fbe3c43688dbd71bcfd0e31a) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/sasl_protocol.cc > libhdfs++: SASL events should be scoped closer to usage > --- > > Key: HDFS-11767 > URL: https://issues.apache.org/jira/browse/HDFS-11767 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11767.HDFS-8707.000.patch > > > "SASL Start" and "SASL End" are called in the SaslProtocol constructor and > destructor. SaslProtocol is held by a shared_ptr so there's a race where the > RpcConnection can be torn down and another can be put in it's place while a > callback is still holding the shared_ptr preventing "SASL End" from being > called. > This can matter a lot because both Cyrus SASL and gsasl aren't thread safe > during negotiation. These event hooks provide a mechanism to pick up a > process wide lock that guards access to the sasl library. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11758) libhdfs++: Catch exceptions thrown by runtime hooks
[ https://issues.apache.org/jira/browse/HDFS-11758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16410464#comment-16410464 ] Hudson commented on HDFS-11758: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13869 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13869/]) HDFS-11758: libhdfs++: Catch exceptions thrown by runtime hooks. (james.clampffer: rev c2386bc2f33d0bec984d31bc5f599c622732e751) * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/bad_datanode_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/bindings/c/hdfs.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/libhdfs_events_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/rpc/rpc_connection_impl.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/rpc_engine_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/include/hdfspp/events.h * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_ext_test.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/reader/block_reader.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/fs/filehandle.cc * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/common/libhdfs_events_impl.cc > libhdfs++: Catch exceptions thrown by runtime hooks > --- > > Key: HDFS-11758 > URL: https://issues.apache.org/jira/browse/HDFS-11758 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: James Clampffer >Assignee: James Clampffer >Priority: Major > Attachments: HDFS-11758.HDFS-8707.000.patch, > HDFS-11758.HDFS-8707.001.patch > > > Right now there's nothing to stop a user supplied event callback from > throwing and unwinding the stack on the asio worker thread. This can lead to > async filesystem operations that never complete because the stack was unwound > before setting up the next async io call. > Best practice is to be really careful about what is done inside event hooks > and callbacks: don't throw, sleep, or anything else that might block. I've > seen a few issues pop up because a hook was made for monitoring purposes and > later on stuff that was able to throw was added without realizing where the > exception may end up. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
