[jira] [Commented] (HDFS-13240) RBF: Update some inaccurate document descriptions
[
https://issues.apache.org/jira/browse/HDFS-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390889#comment-16390889
]
genericqa commented on HDFS-13240:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
25m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 50s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13240 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913543/HDFS-13240.001.patch |
| Optional Tests | asflicense mvnsite |
| uname | Linux 067c7ac2d0ab 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 583f459 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 409 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23348/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Update some inaccurate document descriptions
> -
>
> Key: HDFS-13240
> URL: https://issues.apache.org/jira/browse/HDFS-13240
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Yiqun Lin
>Assignee: Yiqun Lin
>Priority: Minor
> Attachments: HDFS-13240.001.patch
>
>
> In RBF doc, there are some places not describing accurately
> (https://issues.apache.org/jira/browse/HDFS-13214?focusedCommentId=16389026&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16389026).
> This will mislead users sometimes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13247) FileInputStream not closed when read the checksum header from the meta file
[
https://issues.apache.org/jira/browse/HDFS-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390888#comment-16390888
]
liaoyuxiangqin commented on HDFS-13247:
---
[~jzhuge] Thanks for you advise on this!
> FileInputStream not closed when read the checksum header from the meta file
> ---
>
> Key: HDFS-13247
> URL: https://issues.apache.org/jira/browse/HDFS-13247
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Priority: Major
>
> When i read the computeChecksum() of FsDatasetImpl class in datanode, I found
> the following code not closed fis after read the checksum header from the
> meta file, According to the comments of function readDataChecksum() we must
> close inputStream by the caller. Thanks.
> {code:java|title=FsDatasetImpl.java|borderStyle=solid}
> DataChecksum checksum;
> try (FileInputStream fis =
> srcReplica.getFileIoProvider().getFileInputStream(
> srcReplica.getVolume(), srcMeta)) {
> checksum = BlockMetadataHeader.readDataChecksum(
> fis, DFSUtilClient.getIoFileBufferSize(conf), srcMeta);
> }
> final byte[] data = new byte[1 << 16];
> final byte[] crcs = new byte[checksum.getChecksumSize(data.length)];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13247) FileInputStream not closed when read the checksum header from the meta file
[
https://issues.apache.org/jira/browse/HDFS-13247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liaoyuxiangqin resolved HDFS-13247.
---
Resolution: Not A Bug
> FileInputStream not closed when read the checksum header from the meta file
> ---
>
> Key: HDFS-13247
> URL: https://issues.apache.org/jira/browse/HDFS-13247
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Priority: Major
>
> When i read the computeChecksum() of FsDatasetImpl class in datanode, I found
> the following code not closed fis after read the checksum header from the
> meta file, According to the comments of function readDataChecksum() we must
> close inputStream by the caller. Thanks.
> {code:java|title=FsDatasetImpl.java|borderStyle=solid}
> DataChecksum checksum;
> try (FileInputStream fis =
> srcReplica.getFileIoProvider().getFileInputStream(
> srcReplica.getVolume(), srcMeta)) {
> checksum = BlockMetadataHeader.readDataChecksum(
> fis, DFSUtilClient.getIoFileBufferSize(conf), srcMeta);
> }
> final byte[] data = new byte[1 << 16];
> final byte[] crcs = new byte[checksum.getChecksumSize(data.length)];
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390904#comment-16390904
]
genericqa commented on HDFS-11481:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
59s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.6.0 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
10s{color} | {color:red} root in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} branch-2.6.0 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK v1.8.0_151.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
8s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 8s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
Image:yetus/hadoop:date2018-03-08 |
| JIRA Issue | HDFS-11481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856755/HDFS-11481-branch-2.6.0.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 7783a2a0eee1 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.6.0 / e34
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390944#comment-16390944
]
genericqa commented on HDFS-11481:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.6.0 Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m
11s{color} | {color:red} root in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
47s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} branch-2.6.0 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in branch-2.6.0 failed with JDK
v9-internal. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
12s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
9s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_151.
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
10s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v9-internal.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 10s{color}
| {color:red} hadoop-hdfs in the patch failed with JDK v9-internal. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce
Image:yetus/hadoop:date2018-03-08 |
| JIRA Issue | HDFS-11481 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12856755/HDFS-11481-branch-2.6.0.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 0536aa3c97d4 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.6.0 /
[jira] [Commented] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390962#comment-16390962
]
genericqa commented on HDFS-13243:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m
5s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 38s{color} | {color:orange} root: The patch generated 73 new + 670 unchanged
- 2 fixed = 743 total (was 672) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 16s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
30s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
41s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}243m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13243 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913521/HDFS-13243-v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs chec
[jira] [Created] (HDFS-13248) RBF: namenode need to choose block location for the client
Weiwei Wu created HDFS-13248: Summary: RBF: namenode need to choose block location for the client Key: HDFS-13248 URL: https://issues.apache.org/jira/browse/HDFS-13248 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Weiwei Wu When execute a put operation via router, the namenode will choose block location for the router, not for the real client. This will affect the file's locality. I think on both namennode and router, we should add a new addBlock method, or add a parameter for the current addBlock method, to pass the real client information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390975#comment-16390975
]
genericqa commented on HDFS-13246:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}186m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.server.blockmanagement.TestBlockManager |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13246 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913531/HDFS-13246.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux a78498432c3b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 583f459 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23347/artifact/o
[jira] [Updated] (HDFS-13233) RBF: getMountPoint doesn't return the correct mount point of the mount table
[ https://issues.apache.org/jira/browse/HDFS-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhiyuan updated HDFS-13233: --- Attachment: HDFS-13233.004.patch > RBF: getMountPoint doesn't return the correct mount point of the mount table > > > Key: HDFS-13233 > URL: https://issues.apache.org/jira/browse/HDFS-13233 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.2.0 >Reporter: wangzhiyuan >Assignee: wangzhiyuan >Priority: Major > Attachments: HDFS-13233.001.patch, HDFS-13233.002.patch, > HDFS-13233.003.patch, HDFS-13233.004.patch > > > Method MountTableResolver->getMountPoint will traverse mount table and return > the first mount point which the input path starts with, but the condition is > not sufficient. > Suppose the mount point table contains: "/user" "/user/test" "/user/test1", > if the input path is "/user/test111", the return mount point is > "/user/test1", but the correct one should be "/user". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13233) RBF: getMountPoint doesn't return the correct mount point of the mount table
[ https://issues.apache.org/jira/browse/HDFS-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391011#comment-16391011 ] wangzhiyuan commented on HDFS-13233: [~elgoiri] Patch 4 has been uploaded, the failed test case of TestRouterQuota has an issue, Patch 4 fix it > RBF: getMountPoint doesn't return the correct mount point of the mount table > > > Key: HDFS-13233 > URL: https://issues.apache.org/jira/browse/HDFS-13233 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 3.2.0 >Reporter: wangzhiyuan >Assignee: wangzhiyuan >Priority: Major > Attachments: HDFS-13233.001.patch, HDFS-13233.002.patch, > HDFS-13233.003.patch, HDFS-13233.004.patch > > > Method MountTableResolver->getMountPoint will traverse mount table and return > the first mount point which the input path starts with, but the condition is > not sufficient. > Suppose the mount point table contains: "/user" "/user/test" "/user/test1", > if the input path is "/user/test111", the return mount point is > "/user/test1", but the correct one should be "/user". -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13209) DistributedFileSystem.create should allow an option to provide StoragePolicy
[ https://issues.apache.org/jira/browse/HDFS-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391040#comment-16391040 ] Jean-Marc Spaggiari commented on HDFS-13209: Hi [~rakeshr] thanks a lot for your feedback. I guess this will totally work for me. Like you said, this small add to the API will be nice, but maybe not 100% required. I will try with what you proposed here. Thanks. > DistributedFileSystem.create should allow an option to provide StoragePolicy > > > Key: HDFS-13209 > URL: https://issues.apache.org/jira/browse/HDFS-13209 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs >Affects Versions: 3.0.0 >Reporter: Jean-Marc Spaggiari >Priority: Major > > DistributedFileSystem.create allows to get a FSDataOutputStream. The stored > file and related blocks will used the directory based StoragePolicy. > > However, sometime, we might need to keep all files in the same directory > (consistency constraint) but might want some of them on SSD (small, in my > case) until they are processed and merger/removed. Then they will go on the > default policy. > > When creating a file, it will be useful to have an option to specify a > different StoragePolicy... -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13212) RBF: Fix router location cache issue
[
https://issues.apache.org/jira/browse/HDFS-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391069#comment-16391069
]
genericqa commented on HDFS-13212:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 12s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13212 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913557/HDFS-13212-007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 1ad284606808 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4cc9a6d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23351/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23351/testReport/ |
| Max. process+thread count | 4445 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23351/console |
| Powered by | A
[jira] [Commented] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391178#comment-16391178
]
Wei-Chiu Chuang commented on HDFS-13243:
Hi [~gzh1992n] thanks very much for reporting the issue. The NN log is useful
too.
I looked at the patch & log and tried to understand where the problem is. I
don't think I understand the problem fully, but here are some of the thoughts
that I'd like to share with you.
# It seems to me the root of problem is that client would call fsync() with an
incorrect length (shorter than what it is supposed to sync). If that's the case
you should fix the client (DFSOutputStream), rather than the NameNode.
# Looking at the patch, I would like to ask you to stay away from using
reflection. You could refactor FSNamesystem and DFSOutputStream to return a new
FSNamesystem/DFSOutputStream object and override them in the test code. That
way, you don't need to introduce new configurations too. And it'll be much
cleaner.
# I don't understand the following code.
## if lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes() is
unexpected. You should not just log a debug message and ignore it. It's got to
be a WARN level message. You should also log the size of b.getNumBytes() as
well. There's also a grammatical error in the log message too.
## If your fix is correct, you should update the assertion in
FileUnderConstructionFeature#updateLengthOfLastBlock() so it expects neither
COMMITTED nor COMPLETE.
## What should it do when block state is unexpected? I don't think you should
just ignore it.
{code:java}
BlockInfo b = pendingFile.getLastBlock();
if (lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes()) {
LOG.debug("lastBlockLength(" + lastBlockLength + ") seems wrong, maybe have a
bug here?");
return;
}
if (b.getBlockUCState() != BlockUCState.COMMITTED &&
b.getBlockUCState() != BlockUCState.COMPLETE) {{code}
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: HDFS-13243-v1.patch, HDFS-13243-v2.patch
>
>
> HDFS File might get broken because of corrupt block(s) that could be produced
> by calling close and sync in the same time.
> When calling close was not successful, UCBlock status would change to
> COMMITTED, and if a sync request gets popped from queue and processed, sync
> operation would change the last block length.
> After that, DataNode would report all received block to NameNode, and will
> check Block length of all COMMITTED Blocks. But the block length was already
> different between recorded in NameNode memory and reported by DataNode, and
> consequently, the last block is marked as corruptted because of inconsistent
> length.
>
> {panel:title=Log in my hdfs}
> 2018-03-05 04:05:39,261 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocate blk_1085498930_11758129\{UCState=UNDER_CONSTRUCTION,
> truncateBlock=null, primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> for
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,760 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> fsync:
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> for DFSClient_NONMAPREDUCE_1077513762_1
> 2018-03-05 04:05:39,761 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 2) in
> file
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK* addStoredB
[jira] [Comment Edited] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391178#comment-16391178
]
Wei-Chiu Chuang edited comment on HDFS-13243 at 3/8/18 12:37 PM:
-
Hi [~gzh1992n] thanks very much for reporting the issue. The NN log is useful
too.
I looked at the patch & log and tried to understand where the problem is. I
don't think I understand the problem fully, but here are some of the thoughts
that I'd like to share with you.
# It seems to me the root of problem is that client would call fsync() with an
incorrect length (shorter than what it is supposed to sync). If that's the case
you should fix the client (DFSOutputStream), rather than the NameNode.
# Looking at the log, your minimal replication number is 2, rather than 1.
That's very unusual. In my past experience a lot of weird behavior like this
could arise when you have that kind of configuration.
# Looking at the patch, I would like to ask you to stay away from using
reflection. You could refactor FSNamesystem and DFSOutputStream to return a new
FSNamesystem/DFSOutputStream object and override them in the test code. That
way, you don't need to introduce new configurations too. And it'll be much
cleaner.
# I don't understand the following code.
## if lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes() is
unexpected. You should not just log a debug message and ignore it. It's got to
be a WARN level message. You should also log the size of b.getNumBytes() as
well. There's also a grammatical error in the log message too.
## If your fix is correct, you should update the assertion in
FileUnderConstructionFeature#updateLengthOfLastBlock() so it expects neither
COMMITTED nor COMPLETE.
## What should it do when block state is unexpected? I don't think you should
just ignore it.
{code:java}
BlockInfo b = pendingFile.getLastBlock();
if (lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes()) {
LOG.debug("lastBlockLength(" + lastBlockLength + ") seems wrong, maybe have a
bug here?");
return;
}
if (b.getBlockUCState() != BlockUCState.COMMITTED &&
b.getBlockUCState() != BlockUCState.COMPLETE) {{code}
was (Author: jojochuang):
Hi [~gzh1992n] thanks very much for reporting the issue. The NN log is useful
too.
I looked at the patch & log and tried to understand where the problem is. I
don't think I understand the problem fully, but here are some of the thoughts
that I'd like to share with you.
# It seems to me the root of problem is that client would call fsync() with an
incorrect length (shorter than what it is supposed to sync). If that's the case
you should fix the client (DFSOutputStream), rather than the NameNode.
# Looking at the patch, I would like to ask you to stay away from using
reflection. You could refactor FSNamesystem and DFSOutputStream to return a new
FSNamesystem/DFSOutputStream object and override them in the test code. That
way, you don't need to introduce new configurations too. And it'll be much
cleaner.
# I don't understand the following code.
## if lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes() is
unexpected. You should not just log a debug message and ignore it. It's got to
be a WARN level message. You should also log the size of b.getNumBytes() as
well. There's also a grammatical error in the log message too.
## If your fix is correct, you should update the assertion in
FileUnderConstructionFeature#updateLengthOfLastBlock() so it expects neither
COMMITTED nor COMPLETE.
## What should it do when block state is unexpected? I don't think you should
just ignore it.
{code:java}
BlockInfo b = pendingFile.getLastBlock();
if (lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes()) {
LOG.debug("lastBlockLength(" + lastBlockLength + ") seems wrong, maybe have a
bug here?");
return;
}
if (b.getBlockUCState() != BlockUCState.COMMITTED &&
b.getBlockUCState() != BlockUCState.COMPLETE) {{code}
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: HDFS-13243-v1.patch, HDFS-13243-v2.patch
>
>
> HDFS File might get broken because of corrupt block(s) that could be produced
> by calling close and sync in the same time.
> When calling close was not successful, UCBlock status would change to
> COMMITTED, and if a sync request gets popped from queue and processed, sync
> operation would change the last block length.
> After that, DataNode would report all received block to NameNo
[jira] [Comment Edited] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391178#comment-16391178
]
Wei-Chiu Chuang edited comment on HDFS-13243 at 3/8/18 12:39 PM:
-
Hi [~gzh1992n] thanks very much for reporting the issue. The NN log is useful
too.
I looked at the patch & log and tried to understand where the problem is. I
don't think I understand the problem fully, but here are some of the thoughts
that I'd like to share with you.
# It seems to me the root of problem is that client would call fsync() with an
incorrect length (shorter than what it is supposed to sync). If that's the case
you should fix the client (DFSOutputStream), rather than the NameNode.
# Looking at the log, your minimal replication number is 2, rather than 1.
That's very unusual. In my past experience a lot of weird behavior like this
could arise when you have that kind of configuration.
# And why is close() in the picture? IMHO you don't even need to close().
Suppose you block DataNode heartbeat, and let client keep the file open and
then call sync(), the last block's state remains in COMMITTED. Would that cause
the same behavior?
# Looking at the patch, I would like to ask you to stay away from using
reflection. You could refactor FSNamesystem and DFSOutputStream to return a new
FSNamesystem/DFSOutputStream object and override them in the test code. That
way, you don't need to introduce new configurations too. And it'll be much
cleaner.
# I don't understand the following code.
## if lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes() is
unexpected. You should not just log a debug message and ignore it. It's got to
be a WARN level message. You should also log the size of b.getNumBytes() as
well. There's also a grammatical error in the log message too.
## If your fix is correct, you should update the assertion in
FileUnderConstructionFeature#updateLengthOfLastBlock() so it expects neither
COMMITTED nor COMPLETE.
## What should it do when block state is unexpected? I don't think you should
just ignore it.
{code:java}
BlockInfo b = pendingFile.getLastBlock();
if (lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes()) {
LOG.debug("lastBlockLength(" + lastBlockLength + ") seems wrong, maybe have a
bug here?");
return;
}
if (b.getBlockUCState() != BlockUCState.COMMITTED &&
b.getBlockUCState() != BlockUCState.COMPLETE) {{code}
was (Author: jojochuang):
Hi [~gzh1992n] thanks very much for reporting the issue. The NN log is useful
too.
I looked at the patch & log and tried to understand where the problem is. I
don't think I understand the problem fully, but here are some of the thoughts
that I'd like to share with you.
# It seems to me the root of problem is that client would call fsync() with an
incorrect length (shorter than what it is supposed to sync). If that's the case
you should fix the client (DFSOutputStream), rather than the NameNode.
# Looking at the log, your minimal replication number is 2, rather than 1.
That's very unusual. In my past experience a lot of weird behavior like this
could arise when you have that kind of configuration.
# Looking at the patch, I would like to ask you to stay away from using
reflection. You could refactor FSNamesystem and DFSOutputStream to return a new
FSNamesystem/DFSOutputStream object and override them in the test code. That
way, you don't need to introduce new configurations too. And it'll be much
cleaner.
# I don't understand the following code.
## if lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes() is
unexpected. You should not just log a debug message and ignore it. It's got to
be a WARN level message. You should also log the size of b.getNumBytes() as
well. There's also a grammatical error in the log message too.
## If your fix is correct, you should update the assertion in
FileUnderConstructionFeature#updateLengthOfLastBlock() so it expects neither
COMMITTED nor COMPLETE.
## What should it do when block state is unexpected? I don't think you should
just ignore it.
{code:java}
BlockInfo b = pendingFile.getLastBlock();
if (lastBlockLength <= 0 || lastBlockLength <= b.getNumBytes()) {
LOG.debug("lastBlockLength(" + lastBlockLength + ") seems wrong, maybe have a
bug here?");
return;
}
if (b.getBlockUCState() != BlockUCState.COMMITTED &&
b.getBlockUCState() != BlockUCState.COMPLETE) {{code}
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Priority: Critical
> Fix For: 3.2.0
>
[jira] [Assigned] (HDFS-13243) Get CorruptBlock because of calling close and sync in same time
[
https://issues.apache.org/jira/browse/HDFS-13243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDFS-13243:
--
Assignee: Zephyr Guo
> Get CorruptBlock because of calling close and sync in same time
> ---
>
> Key: HDFS-13243
> URL: https://issues.apache.org/jira/browse/HDFS-13243
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.7.2, 3.2.0
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: HDFS-13243-v1.patch, HDFS-13243-v2.patch
>
>
> HDFS File might get broken because of corrupt block(s) that could be produced
> by calling close and sync in the same time.
> When calling close was not successful, UCBlock status would change to
> COMMITTED, and if a sync request gets popped from queue and processed, sync
> operation would change the last block length.
> After that, DataNode would report all received block to NameNode, and will
> check Block length of all COMMITTED Blocks. But the block length was already
> different between recorded in NameNode memory and reported by DataNode, and
> consequently, the last block is marked as corruptted because of inconsistent
> length.
>
> {panel:title=Log in my hdfs}
> 2018-03-05 04:05:39,261 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> allocate blk_1085498930_11758129\{UCState=UNDER_CONSTRUCTION,
> truncateBlock=null, primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> for
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,760 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> fsync:
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> for DFSClient_NONMAPREDUCE_1077513762_1
> 2018-03-05 04:05:39,761 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 0 < minimum = 2) in
> file
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com%2C16020%2C1519845790686.default.1520193926515
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK* addStoredBlock:
> blockMap updated: 10.0.0.220:50010 is added to
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> size 2054413
> 2018-03-05 04:05:39,761 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as corrupt on
> 10.0.0.219:50010 by
> hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com/10.0.0.219 because block is
> COMMITTED and reported length 2054413 does not match length in block map
> 141232
> 2018-03-05 04:05:39,762 INFO BlockStateChange: BLOCK
> NameSystem.addToCorruptReplicasMap: blk_1085498930 added as corrupt on
> 10.0.0.218:50010 by
> hb-j5e517al6xib80rkb-004.hbase.rds.aliyuncs.com/10.0.0.218 because block is
> COMMITTED and reported length 2054413 does not match length in block map
> 141232
> 2018-03-05 04:05:40,162 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
> blk_1085498930_11758129\{UCState=COMMITTED, truncateBlock=null,
> primaryNodeIndex=-1,
> replicas=[ReplicaUC[[DISK]DS-32c7e479-3845-4a44-adf1-831edec7506b:NORMAL:10.0.0.219:50010|RBW],
>
> ReplicaUC[[DISK]DS-a9a5d653-c049-463d-8e4a-d1f0dc14409c:NORMAL:10.0.0.220:50010|RBW],
>
> ReplicaUC[[DISK]DS-f2b7c04a-b724-4c69-abbf-d2e416f70706:NORMAL:10.0.0.218:50010|RBW]]}
> is not COMPLETE (ucState = COMMITTED, replication# = 3 >= minimum = 2) in
> file
> /hbase/WALs/hb-j5e517al6xib80rkb-006.hbase.rds.aliyuncs.com,16020,1519845790686/hb-j5e517al6xib80rkb-006.hbase.rds.aliy
[jira] [Commented] (HDFS-13233) RBF: getMountPoint doesn't return the correct mount point of the mount table
[
https://issues.apache.org/jira/browse/HDFS-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391210#comment-16391210
]
genericqa commented on HDFS-13233:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13233 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913576/HDFS-13233.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6c30fce47bf8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4cc9a6d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23353/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23353/testReport/ |
| Max. process+thread count | 3844 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/h
[jira] [Commented] (HDFS-13246) FileInputStream redundant closes in readReplicasFromCache
[
https://issues.apache.org/jira/browse/HDFS-13246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391228#comment-16391228
]
Rushabh S Shah commented on HDFS-13246:
---
Thanks [~liaoyuxiangqin] for finding this.
+1 non-binding.
> FileInputStream redundant closes in readReplicasFromCache
> --
>
> Key: HDFS-13246
> URL: https://issues.apache.org/jira/browse/HDFS-13246
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: liaoyuxiangqin
>Priority: Major
> Attachments: HDFS-13246.001.patch
>
>
> When i read the readReplicasFromCache() of BlockPoolSlice class in datanode,
> I found the following code closes fileinputstream redundant, I think
> IOUtils.closeStream(inputStream) in finally code block could guarantee close
> the inputStream correctly, So the
> inputStream.close() can remove. Thanks.
>
> {code:java|title=BlockPoolSlice.java|borderStyle=solid}
> FileInputStream inputStream = null;
> try {
> inputStream = fileIoProvider.getFileInputStream(volume, replicaFile);
> BlockListAsLongs blocksList =
> BlockListAsLongs.readFrom(inputStream, maxDataLength);
> if (blocksList == null) {
> return false;
> }
> for (BlockReportReplica replica : blocksList) {
> switch (replica.getState()) {
> case FINALIZED:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, true);
> break;
> case RUR:
> case RBW:
> case RWR:
> addReplicaToReplicasMap(replica, tmpReplicaMap,
> lazyWriteReplicaMap, false);
> break;
> default:
> break;
> }
> }
> inputStream.close();
> // Now it is safe to add the replica into volumeMap
> // In case of any exception during parsing this cache file, fall back
> // to scan all the files on disk.
> for (Iterator iter =
> tmpReplicaMap.replicas(bpid).iterator(); iter.hasNext(); ) {
> ReplicaInfo info = iter.next();
> // We use a lightweight GSet to store replicaInfo, we need to remove
> // it from one GSet before adding to another.
> iter.remove();
> volumeMap.add(bpid, info);
> }
> LOG.info("Successfully read replica from cache file : "
> + replicaFile.getPath());
> return true;
> } catch (Exception e) {
> // Any exception we need to revert back to read from disk
> // Log the error and return false
> LOG.info("Exception occurred while reading the replicas cache file: "
> + replicaFile.getPath(), e );
> return false;
> }
> finally {
> if (!fileIoProvider.delete(volume, replicaFile)) {
> LOG.info("Failed to delete replica cache file: " +
> replicaFile.getPath());
> }
> // close the inputStream
> IOUtils.closeStream(inputStream);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13096) HDFS group quota
[
https://issues.apache.org/jira/browse/HDFS-13096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391236#comment-16391236
]
He Xiaoqiao commented on HDFS-13096:
[~Tagar], is this issue going on?
> HDFS group quota
>
>
> Key: HDFS-13096
> URL: https://issues.apache.org/jira/browse/HDFS-13096
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, fs, hdfs, nn
>Affects Versions: 2.8.3, 2.7.5, 3.0.0
>Reporter: Ruslan Dautkhanov
>Priority: Major
>
> We have groups of people that have their own set of HDFS directories.
> For example, they have HDFS staging place for new files:
> /datascience
> /analysts
> ...
> but at the same time they have Hive warehouse directory
> /hivewarehouse/datascience
> /hivewarehouse/analysts
> ...
> on top of that they also have some files stored under /user/${username}/
> It's always been a challenge to maintain a combined quota on all HDFS
> locations a particular group of people owns. As we're currently forced to put
> a particular quota for each directory independently.
> It would be great if HDFS would have a quota tied either
> - to a set of HDFS locations ;
> - or to a group of people (where `group`is defined as which HDFS group a
> particular file/directory belongs to).
> Linux allows to define quotas at group level, i.e. `edquota -g devel` etc..
> would be great to have the same at HDFS level.
> Other thoughts and ideas?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391326#comment-16391326
]
Yongjun Zhang commented on HDFS-11481:
--
Thanks [~mavinmar...@gmail.com] for working on this issue and [~andrew.wang]
for the review.
Hi [~jingzhao],
Would you please help answering Andrew's question in his comments above:
[~jingzhao] is this the right direction? Not sure why these methods don't use
{{resolvePath}}.
Thanks a lot.
> hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
> ---
>
> Key: HDFS-11481
> URL: https://issues.apache.org/jira/browse/HDFS-11481
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
>Priority: Minor
> Attachments: HDFS-11481-branch-2.6.0.001.patch, HDFS-11481.001.patch,
> HDFS-11481.002.patch
>
>
> Successful command:
> {code}
> #> hdfs snapshotDiff /tmp/dir s1 s2
> Difference between snapshot s1 and snapshot s2 under directory /tmp/dir:
> M .
> + ./file1.txt
> {code}
> Unsuccessful command:
> {code}
> #> hdfs snapshotDiff /.reserved/raw/tmp/dir s1 s2
> snapshotDiff: Directory does not exist: /.reserved/raw/tmp/dir
> {code}
> Prefixing with raw path should run successfully and return same output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
maobaolong updated HDFS-13226:
--
Attachment: HDFS-13226.005.patch
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391352#comment-16391352
]
maobaolong commented on HDFS-13226:
---
[~linyiqun] I've upload a new patch just now, not sure i have addressed all
your comments.
* I update the invalidate method java doc, remove the return value's
description.
* Create a new test case in TestMountTable.
* I keep the start with "//" check, and another Jira will complete the check
rule of mount entry.
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[
https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391376#comment-16391376
]
maobaolong commented on HDFS-13241:
---
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code:bash}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
> RBF: TestRouterSafemode failed if the port is in use
> -
>
> Key: HDFS-13241
> URL: https://issues.apache.org/jira/browse/HDFS-13241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, test
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13241.001.patch
>
>
> TestRouterSafemode failed if the port is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[
https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391376#comment-16391376
]
maobaolong edited comment on HDFS-13241 at 3/8/18 3:13 PM:
---
[~elgoiri]
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
was (Author: maobaolong):
I use a tool start a tcp server and bind the port , then, i run the test
all of rbf, the test are all passed. So i think the others RBF tests is good
now.
{code:bash}
mvn -Dtest=org.apache.hadoop.hdfs.server.federation.** test
...
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 06:00 min
[INFO] Finished at: 2018-03-08T23:08:28+08:00
[INFO] Final Memory: 163M/1531M
[INFO]
{code}
> RBF: TestRouterSafemode failed if the port is in use
> -
>
> Key: HDFS-13241
> URL: https://issues.apache.org/jira/browse/HDFS-13241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, test
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13241.001.patch
>
>
> TestRouterSafemode failed if the port is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391383#comment-16391383 ] maobaolong commented on HDFS-13248: --- Maybe we can modify the client logic, let the client get the correct namenode from router, then the client call to namenode. Not sure this is a wise way, because people doesn't want to update client package, further more, some body doesn't use the client. > RBF: namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Weiwei Wu >Priority: Major > > When execute a put operation via router, the namenode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both namennode and router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[ https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391387#comment-16391387 ] maobaolong commented on HDFS-12615: --- [~elgoiri] I've create a new issue to track the dbms state store. BTW, I have a question, should us config the remote ns or data center's router to client? > Router-based HDFS federation phase 2 > > > Key: HDFS-12615 > URL: https://issues.apache.org/jira/browse/HDFS-12615 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Labels: RBF > > This umbrella JIRA tracks set of improvements over the Router-based HDFS > federation (HDFS-10467). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13240) RBF: Update some inaccurate document descriptions
[ https://issues.apache.org/jira/browse/HDFS-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391482#comment-16391482 ] Wei Yan commented on HDFS-13240: Some minors: * For the table dfs.federation.router.file.resolver.client.class, would be better to wrap the classes by character `, same as the dfs.federation.router.store.serializer one. * "The subclusters in RBF are not required to be the independent HDFS clusters, a normal federation cluster or a mixed federation cluster is also allowed." This one looks confused.. you mean the underlying subclusters can also be the original HDFS federation (with multiple pools)? > RBF: Update some inaccurate document descriptions > - > > Key: HDFS-13240 > URL: https://issues.apache.org/jira/browse/HDFS-13240 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Yiqun Lin >Assignee: Yiqun Lin >Priority: Minor > Attachments: HDFS-13240.001.patch > > > In RBF doc, there are some places not describing accurately > (https://issues.apache.org/jira/browse/HDFS-13214?focusedCommentId=16389026&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16389026). > This will mislead users sometimes. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13186) [WRITE] Multipart Multinode uploader API + Implementations
[
https://issues.apache.org/jira/browse/HDFS-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-13186:
--
Attachment: HDFS-13186.patch
> [WRITE] Multipart Multinode uploader API + Implementations
> --
>
> Key: HDFS-13186
> URL: https://issues.apache.org/jira/browse/HDFS-13186
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Priority: Major
> Attachments: HDFS-13186.patch
>
>
> To write files in parallel to an external storage system as in HDFS-12090,
> there are two approaches:
> # Naive approach: use a single datanode per file that copies blocks locally
> as it streams data to the external service. This requires a copy for each
> block inside the HDFS system and then a copy for the block to be sent to the
> external system.
> # Better approach: Single point (e.g. Namenode or SPS style external client)
> and Datanodes coordinate in a multipart - multinode upload.
> This system needs to work with multiple back ends and needs to coordinate
> across the network. So we propose an API that resembles the following:
> {code:java}
> public UploadHandle multipartInit(Path filePath) throws IOException;
> public PartHandle multipartPutPart(InputStream inputStream,
> int partNumber, UploadHandle uploadId) throws IOException;
> public void multipartComplete(Path filePath,
> List> handles,
> UploadHandle multipartUploadId) throws IOException;{code}
> Here, UploadHandle and PartHandle are opaque handlers in the vein of
> PathHandle so they can be serialized and deserialized in hadoop-hdfs project
> without knowledge of how to deserialize e.g. S3A's version of a UpoadHandle
> and PartHandle.
> In an object store such as S3A, the implementation is straight forward. In
> the case of writing multipart/multinode to HDFS, we can write each block as a
> file part. The complete call will perform a concat on the blocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13186) [WRITE] Multipart Multinode uploader API + Implementations
[
https://issues.apache.org/jira/browse/HDFS-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewan Higgs updated HDFS-13186:
--
Status: Patch Available (was: Open)
Submitting a patch with an abstract class for {{MultipartUploader}} and three
implementations: {{LocalMultipartUploader}}, {{DFSMultipartUploader}}, and
{{S3AMultipartUploader}}.
> [WRITE] Multipart Multinode uploader API + Implementations
> --
>
> Key: HDFS-13186
> URL: https://issues.apache.org/jira/browse/HDFS-13186
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Priority: Major
> Attachments: HDFS-13186.patch
>
>
> To write files in parallel to an external storage system as in HDFS-12090,
> there are two approaches:
> # Naive approach: use a single datanode per file that copies blocks locally
> as it streams data to the external service. This requires a copy for each
> block inside the HDFS system and then a copy for the block to be sent to the
> external system.
> # Better approach: Single point (e.g. Namenode or SPS style external client)
> and Datanodes coordinate in a multipart - multinode upload.
> This system needs to work with multiple back ends and needs to coordinate
> across the network. So we propose an API that resembles the following:
> {code:java}
> public UploadHandle multipartInit(Path filePath) throws IOException;
> public PartHandle multipartPutPart(InputStream inputStream,
> int partNumber, UploadHandle uploadId) throws IOException;
> public void multipartComplete(Path filePath,
> List> handles,
> UploadHandle multipartUploadId) throws IOException;{code}
> Here, UploadHandle and PartHandle are opaque handlers in the vein of
> PathHandle so they can be serialized and deserialized in hadoop-hdfs project
> without knowledge of how to deserialize e.g. S3A's version of a UpoadHandle
> and PartHandle.
> In an object store such as S3A, the implementation is straight forward. In
> the case of writing multipart/multinode to HDFS, we can write each block as a
> file part. The complete call will perform a concat on the blocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391485#comment-16391485
]
Wei Yan commented on HDFS-13230:
+1. [~elgoiri] any further comments?
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13186) [WRITE] Multipart Multinode uploader API + Implementations
[
https://issues.apache.org/jira/browse/HDFS-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391495#comment-16391495
]
Ewan Higgs commented on HDFS-13186:
---
[~ste...@apache.org]
{quote} * I worry about how to do ser/deser securely; because I don't want to
use things like Java serialization to persist the intermediate state
* the multipart put should also allow caller to supply a File and range refs;
makes it easier to upload buffered data, especially when client libs (like AWS
SDK) are better at recovery of POST failures when it knows its reading off a
file.{quote}
I'm not sure if/how these concerns are applicable here. Maybe with the given
patch the issues could be described more concretely? Thanks!
> [WRITE] Multipart Multinode uploader API + Implementations
> --
>
> Key: HDFS-13186
> URL: https://issues.apache.org/jira/browse/HDFS-13186
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ewan Higgs
>Priority: Major
> Attachments: HDFS-13186.patch
>
>
> To write files in parallel to an external storage system as in HDFS-12090,
> there are two approaches:
> # Naive approach: use a single datanode per file that copies blocks locally
> as it streams data to the external service. This requires a copy for each
> block inside the HDFS system and then a copy for the block to be sent to the
> external system.
> # Better approach: Single point (e.g. Namenode or SPS style external client)
> and Datanodes coordinate in a multipart - multinode upload.
> This system needs to work with multiple back ends and needs to coordinate
> across the network. So we propose an API that resembles the following:
> {code:java}
> public UploadHandle multipartInit(Path filePath) throws IOException;
> public PartHandle multipartPutPart(InputStream inputStream,
> int partNumber, UploadHandle uploadId) throws IOException;
> public void multipartComplete(Path filePath,
> List> handles,
> UploadHandle multipartUploadId) throws IOException;{code}
> Here, UploadHandle and PartHandle are opaque handlers in the vein of
> PathHandle so they can be serialized and deserialized in hadoop-hdfs project
> without knowledge of how to deserialize e.g. S3A's version of a UpoadHandle
> and PartHandle.
> In an object store such as S3A, the implementation is straight forward. In
> the case of writing multipart/multinode to HDFS, we can write each block as a
> file part. The complete call will perform a concat on the blocks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13234) Remove renew configuration instance in ConfiguredFailoverProxyProvider and reduce memory footprint for client
[
https://issues.apache.org/jira/browse/HDFS-13234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391519#comment-16391519
]
Kihwal Lee commented on HDFS-13234:
---
[~jlowe] and I discussed a bit about the conf issue this morning. Configuration
has both performance and memory foot print issue, but coming up with a single
generic solution to solve them for all use cases is difficult, if not
impossible. That's one of the road blocks many previous improvement attempts
have met. For use cases that do not require refreshing, we can have a single
mutable instance to load/reload all resources, instead of duplicating for each
config instance. Each new conf can have its own "overlay" map internally to
keep track of locally set keys/values. For the keys not found in this map, it
will look them up in the base instance. The look-ups will get a bit more
expensive, but it avoids problem of multiple resource reloads and object
duplication. Since this might not work well with refreshable configs, it would
be better to make it a new feature (i.e. a new version of ctor) and offer it
opt-in basis. I think most client-side code will be able to take advantage of
this.
Related: HADOOP-11223 and HADOOP-9570
We can start a design/feasibility discussion, if there is enough interest.
> Remove renew configuration instance in ConfiguredFailoverProxyProvider and
> reduce memory footprint for client
> -
>
> Key: HDFS-13234
> URL: https://issues.apache.org/jira/browse/HDFS-13234
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: fs, ha, hdfs-client
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-13234.001.patch
>
>
> The memory footprint of #DFSClient is very considerable in some special
> scenario since there are many #Configuration instances and occupy much memory
> resource (In an extreme case, org.apache.hadoop.conf.Configuration occupies
> over 600MB we meet under HDFS Federation an HA with QJM and there are dozens
> of NameNodes). I think some new Configuration instance is not necessary. Such
> as #ConfiguredFailoverProxyProvider initialization.
> {code:java}
> public ConfiguredFailoverProxyProvider(Configuration conf, URI uri,
> Class xface, HAProxyFactory factory) {
> this.xface = xface;
> this.conf = new Configuration(conf);
> ..
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13248) RBF: namenode need to choose block location for the client
[
https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391560#comment-16391560
]
Íñigo Goiri commented on HDFS-13248:
I've tried to follow the logic in {{NamenodeRpcServer}} and it doesn't look
like it uses the RPC address but the {{clientName}} parameter.
{{NamenodeRpcServer#addBlock}} goes to {{FSNamesystem#getAdditionalBlock}}
which calls {{FSDirWriteFileOp#chooseTargetForNewBlock}}.
Inside {{chooseTargetForNewBlock}}, we call
{{DatanodeManager#getDatanodeByHost()}} with a parameter called
{{clientMachine}} which seems to come from
{{FSDirWriteFileOp#validateAddBlock()}}:
{code}
clientMachine =
pendingFile.getFileUnderConstructionFeature().getClientMachine();
{code}
I'm not 100% sure but I think this value is the parameter {{clientName}} which
we initially pass to {{RouterRpcServer#addBlock}}.
> RBF: namenode need to choose block location for the client
> --
>
> Key: HDFS-13248
> URL: https://issues.apache.org/jira/browse/HDFS-13248
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Weiwei Wu
>Priority: Major
>
> When execute a put operation via router, the namenode will choose block
> location for the router, not for the real client. This will affect the file's
> locality.
> I think on both namennode and router, we should add a new addBlock method, or
> add a parameter for the current addBlock method, to pass the real client
> information.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13248) RBF: namenode need to choose block location for the client
[ https://issues.apache.org/jira/browse/HDFS-13248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13248: --- Description: When execute a put operation via router, the NameNode will choose block location for the router, not for the real client. This will affect the file's locality. I think on both NameNode and Router, we should add a new addBlock method, or add a parameter for the current addBlock method, to pass the real client information. was: When execute a put operation via router, the namenode will choose block location for the router, not for the real client. This will affect the file's locality. I think on both namennode and router, we should add a new addBlock method, or add a parameter for the current addBlock method, to pass the real client information. > RBF: namenode need to choose block location for the client > -- > > Key: HDFS-13248 > URL: https://issues.apache.org/jira/browse/HDFS-13248 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Weiwei Wu >Priority: Major > > When execute a put operation via router, the NameNode will choose block > location for the router, not for the real client. This will affect the file's > locality. > I think on both NameNode and Router, we should add a new addBlock method, or > add a parameter for the current addBlock method, to pass the real client > information. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391573#comment-16391573
]
Íñigo Goiri commented on HDFS-13230:
[^HDFS-13230.001.patch] LGTM.
I would hate to waste a Yetus so clean as this one :)
The unit test passes in 0.53 seconds [link
title|https://builds.apache.org/job/PreCommit-HDFS-Build/23344/testReport/org.apache.hadoop.hdfs.server.federation.router/TestConnectionManager/].
+1
I'd like to build HDFS-13232 on top of this.
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391579#comment-16391579
]
Íñigo Goiri commented on HDFS-13230:
Thanks [~csun] for the contribution and [~ywskycn] for reporting and reviewing.
I committed to trunk, branch-3.1, branch-3.0 and branch-2.9.
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13230:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2.0
3.0.2
2.9.1
3.1.0
Status: Resolved (was: Patch Available)
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13232) RBF: ConnectionPool should return first usable connection
[
https://issues.apache.org/jira/browse/HDFS-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391582#comment-16391582
]
Íñigo Goiri commented on HDFS-13232:
I've already committed HDFS-13230 so this one can build on the unit test added
there and cover the case [~csun] had there.
> RBF: ConnectionPool should return first usable connection
> -
>
> Key: HDFS-13232
> URL: https://issues.apache.org/jira/browse/HDFS-13232
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Ekanth S
>Priority: Minor
>
> In current ConnectionPool.getConnection(), it will return the first active
> connection:
> {code:java}
> for (int i=0; i int index = (threadIndex + i) % size;
> conn = tmpConnections.get(index);
> if (conn != null && !conn.isUsable()) {
> return conn;
> }
> }
> {code}
> Here "!conn.isUsable()" should be "conn.isUsable()".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391586#comment-16391586
]
Chao Sun commented on HDFS-13230:
-
Thanks [~elgoiri] and [~ywskycn] for the review! BTW: I think the commit
message is wrong: it should be HDFS-13230, not HDFS-13232 :)
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391591#comment-16391591
]
genericqa commented on HDFS-13226:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 45s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 4 new + 1 unchanged - 0 fixed = 5 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 18s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.metrics.TestFederationMetrics |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13226 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913614/HDFS-13226.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux f9a32e31d661 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7ef4d94 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23354/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23354/artifact/out/patch-unit-hadoop-hdfs-project_ha
[jira] [Commented] (HDFS-12615) Router-based HDFS federation phase 2
[
https://issues.apache.org/jira/browse/HDFS-12615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391593#comment-16391593
]
Íñigo Goiri commented on HDFS-12615:
{quote}
BTW, I have a question, should us config the remote ns or data center's router
to client?
{quote}
For the client, it's enough to setup the Router addresses as a
{{ConfiguredFailoverProxyProvider}}, no need to setup the remote nameservices.
The documentation
[here|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSRouterFederation.html]
shows the exmaple to access both directly and through the Router but you only
need:
{code}
dfs.nameservices
ns-fed
dfs.namenodes.ns-fed
r1,r2
dfs.namenode.rpc-address.ns-fed.r1
router1:rpc-port
dfs.namenode.rpc-address.ns-fed.r2
router2:rpc-port
dfs.client.failover.proxy.provider.ns-fed
org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.random.order
true
{code}
> Router-based HDFS federation phase 2
>
>
> Key: HDFS-12615
> URL: https://issues.apache.org/jira/browse/HDFS-12615
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Labels: RBF
>
> This umbrella JIRA tracks set of improvements over the Router-based HDFS
> federation (HDFS-10467).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13241) RBF: TestRouterSafemode failed if the port 8888 is in use
[
https://issues.apache.org/jira/browse/HDFS-13241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391596#comment-16391596
]
Íñigo Goiri commented on HDFS-13241:
Surprising that this only affects this unit test.
In that case, [^HDFS-13241.001.patch] LGTM.
A little concerned about {{TestRouterNamenodeMonitoring}} failing
[here|https://builds.apache.org/job/PreCommit-HDFS-Build/23334/testReport/org.apache.hadoop.hdfs.server.federation.router/TestRouterNamenodeMonitoring/testNamenodeMonitoring/].
I would rerun it again.
> RBF: TestRouterSafemode failed if the port is in use
> -
>
> Key: HDFS-13241
> URL: https://issues.apache.org/jira/browse/HDFS-13241
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, test
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Attachments: HDFS-13241.001.patch
>
>
> TestRouterSafemode failed if the port is in use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11481) hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
[
https://issues.apache.org/jira/browse/HDFS-11481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391601#comment-16391601
]
Yongjun Zhang commented on HDFS-11481:
--
Hi [~mavinmar...@gmail.com],
Thanks for your earlier work. The patch doesn't apply in trunk, wonder if you
could produce a new rev and we can review from there?
Thanks.
> hdfs snapshotDiff /.reserved/raw/... fails on snapshottable directories
> ---
>
> Key: HDFS-11481
> URL: https://issues.apache.org/jira/browse/HDFS-11481
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.6.0
>Reporter: Mavin Martin
>Assignee: Mavin Martin
>Priority: Minor
> Attachments: HDFS-11481-branch-2.6.0.001.patch, HDFS-11481.001.patch,
> HDFS-11481.002.patch
>
>
> Successful command:
> {code}
> #> hdfs snapshotDiff /tmp/dir s1 s2
> Difference between snapshot s1 and snapshot s2 under directory /tmp/dir:
> M .
> + ./file1.txt
> {code}
> Unsuccessful command:
> {code}
> #> hdfs snapshotDiff /.reserved/raw/tmp/dir s1 s2
> snapshotDiff: Directory does not exist: /.reserved/raw/tmp/dir
> {code}
> Prefixing with raw path should run successfully and return same output.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13226) RBF: We should throw the failure validate and refuse this mount entry
[
https://issues.apache.org/jira/browse/HDFS-13226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391604#comment-16391604
]
Íñigo Goiri commented on HDFS-13226:
I like the new unit test in [^HDFS-13226.005.patch].
There are a couple nits:
* Check styles
* Still a couple {{Assert.assertEquals}}
> RBF: We should throw the failure validate and refuse this mount entry
> -
>
> Key: HDFS-13226
> URL: https://issues.apache.org/jira/browse/HDFS-13226
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: maobaolong
>Assignee: maobaolong
>Priority: Major
> Labels: RBF
> Fix For: 3.2.0
>
> Attachments: HDFS-13226.001.patch, HDFS-13226.002.patch,
> HDFS-13226.003.patch, HDFS-13226.004.patch, HDFS-13226.005.patch
>
>
> one of the mount entry source path rule is that the source path must start
> with '\', somebody didn't follow the rule and execute the following command:
> {code:bash}
> $ hdfs dfsrouteradmin -add addnode/ ns1 /addnode/
> {code}
> But, the console show we are successful add this entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391607#comment-16391607
]
Wei Yan commented on HDFS-13230:
{quote}I would hate to waste a Yetus so clean as this one
{quote}
Haven't seen a +1 from Yetus for a long time :D
BTW, you forgot branch-2?
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13233) RBF: getMountPoint doesn't return the correct mount point of the mount table
[
https://issues.apache.org/jira/browse/HDFS-13233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391611#comment-16391611
]
Íñigo Goiri commented on HDFS-13233:
Thanks [~striver.wang] for the fix.
The fix to {{TestRouterQuota}} LGTM too.
[~linyiqun] do you mind double chekcing?
+1 from my side waiting for [~linyiqun] confirmation.
> RBF: getMountPoint doesn't return the correct mount point of the mount table
>
>
> Key: HDFS-13233
> URL: https://issues.apache.org/jira/browse/HDFS-13233
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.2.0
>Reporter: wangzhiyuan
>Assignee: wangzhiyuan
>Priority: Major
> Attachments: HDFS-13233.001.patch, HDFS-13233.002.patch,
> HDFS-13233.003.patch, HDFS-13233.004.patch
>
>
> Method MountTableResolver->getMountPoint will traverse mount table and return
> the first mount point which the input path starts with, but the condition is
> not sufficient.
> Suppose the mount point table contains: "/user" "/user/test" "/user/test1",
> if the input path is "/user/test111", the return mount point is
> "/user/test1", but the correct one should be "/user".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13212) RBF: Fix router location cache issue
[ https://issues.apache.org/jira/browse/HDFS-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-13212: -- Assignee: Weiwei Wu > RBF: Fix router location cache issue > > > Key: HDFS-13212 > URL: https://issues.apache.org/jira/browse/HDFS-13212 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: federation, hdfs >Reporter: Weiwei Wu >Assignee: Weiwei Wu >Priority: Major > Attachments: HDFS-13212-001.patch, HDFS-13212-002.patch, > HDFS-13212-003.patch, HDFS-13212-004.patch, HDFS-13212-005.patch, > HDFS-13212-006.patch, HDFS-13212-007.patch > > > The MountTableResolver refreshEntries function have a bug when add a new > mount table entry which already have location cache. The old location cache > will never be invalid until this mount point change again. > Need to invalid the location cache when add the mount table entries. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13236) Standby NN down with error encountered while tailing edits
[
https://issues.apache.org/jira/browse/HDFS-13236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391622#comment-16391622
]
Kihwal Lee commented on HDFS-13236:
---
For block placement differences, {{DFSNetworkTopology}} is new in Hadoop 3. It
might be related. See HDFS-11419. [~vagarychen] might be able to tell whether
it is related.
Missing Client ID is very strange as it has a call ID. Both come from the
handler's thread local variable set by the RPC server. The client ID field
isn't the last field in the edit either. It is created when a RPC client is
created and this is set in the connection context header. It is all internal
and automatic. I don't know what happens when the connection is dropped before
edit logging, while the call is still being processed. [~daryn], does server
reset the client ID in this case?
> Standby NN down with error encountered while tailing edits
> --
>
> Key: HDFS-13236
> URL: https://issues.apache.org/jira/browse/HDFS-13236
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: journal-node, namenode
>Affects Versions: 3.0.0
>Reporter: Yuriy Malygin
>Priority: Major
>
> After update Hadoop from 2.7.3 to 3.0.0 standby NN down with error
> encountered while tailing edits from JN:
> {code:java}
> Feb 28 01:58:31 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:31,594
> INFO [FSImageSaver for /one/hadoop-data/dfs of type IMAGE_AND_EDITS]
> FSImageFormatProtobuf - Image file
> /one/hadoop-data/dfs/current/fsimage.ckpt_012748979
> 98 of size 4595971949 bytes saved in 93 seconds.
> Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445
> INFO [Standby State Checkpointer] NNStorageRetentionManager - Going to retain
> 2 images with txid >= 1274897935
> Feb 28 01:58:33 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:33,445
> INFO [Standby State Checkpointer] NNStorageRetentionManager - Purging old
> image
> FSImageFile(file=/one/hadoop-data/dfs/current/fsimage_01274897875,
> cpktTxId
> =01274897875)
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660
> INFO [Edit log tailer] FSImage - Reading
> org.apache.hadoop.hdfs.server.namenode.RedundantEditLogInputStream@6a168e6f
> expecting start txid #1274897999
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,660
> INFO [Edit log tailer] FSImage - Start loading edits file
> http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup&segmentTxId=1274897999&storageInfo=-64%3A10
> 56233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848&inProgressOk=true,
> http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup&segmentTxId=1274897999&storageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848&;
> inProgressOk=true
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661
> INFO [Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream
> 'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup&segmentTxId=1274897999&s
> torageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848&inProgressOk=true,
>
> http://srve2916.local:8480/getJournal?jid=datalab-hadoop-backup&segmentTxId=1274897999&storageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217
> -aef5-6ed206893848&inProgressOk=true' to transaction ID 1274897999
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,661
> INFO [Edit log tailer] RedundantEditLogInputStream - Fast-forwarding stream
> 'http://srvd87.local:8480/getJournal?jid=datalab-hadoop-backup&segmentTxId=1274897999&storageInfo=-64%3A1056233980%3A0%3ACID-1fba08aa-c8bd-4217-aef5-6ed206893848&inProgressOk=true'
> to transaction ID 1274897999
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: 2018-02-28 01:58:34,680
> ERROR [Edit log tailer] FSEditLogLoader - Encountered exception on operation
> AddOp [length=0, inodeId=145550319,
> path=/kafka/parquet/infrastructureGrace/date=2018-02-28/_temporary/1/_temporary/attempt_1516181147167_20856_r_98_0/part-r-00098.gz.parquet,
> replication=3, mtime=1519772206615, atime=1519772206615,
> blockSize=134217728, blocks=[], permissions=root:supergroup:rw-r--r--,
> aclEntries=null,
> clientName=DFSClient_attempt_1516181147167_20856_r_98_0_1523538799_1,
> clientMachine=10.137.2.142, overwrite=false, RpcClientId=,
> RpcCallId=271996603, storagePolicyId=0, erasureCodingPolicyId=0,
> opCode=OP_ADD, txid=1274898002]
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]:
> java.lang.IllegalArgumentException: Invalid clientId - length is 0 expected
> length 16
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]: at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> Feb 28 01:58:34 srvd2135 datalab-namenode[15566]:
[jira] [Commented] (HDFS-13212) RBF: Fix router location cache issue
[
https://issues.apache.org/jira/browse/HDFS-13212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391628#comment-16391628
]
Íñigo Goiri commented on HDFS-13212:
After a deeper look, the part that concerns me is the following:
{code}
485 // Add the default location and test location cache
486 assertEquals("null->/testlocationcache",
487
mountTable.getDestinationForPath("/testlocationcache").toString());
{code}
The fact that is returning null is weird.
My guess is that the default location is {{null}} but we should have some other
as default and verify that once we remove the entry, we go to that default.
So I would make say 0 the default and then assert we point there.
> RBF: Fix router location cache issue
>
>
> Key: HDFS-13212
> URL: https://issues.apache.org/jira/browse/HDFS-13212
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, hdfs
>Reporter: Weiwei Wu
>Assignee: Weiwei Wu
>Priority: Major
> Attachments: HDFS-13212-001.patch, HDFS-13212-002.patch,
> HDFS-13212-003.patch, HDFS-13212-004.patch, HDFS-13212-005.patch,
> HDFS-13212-006.patch, HDFS-13212-007.patch
>
>
> The MountTableResolver refreshEntries function have a bug when add a new
> mount table entry which already have location cache. The old location cache
> will never be invalid until this mount point change again.
> Need to invalid the location cache when add the mount table entries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13249) Document webhdfs support for getting snapshottable directory list
Lokesh Jain created HDFS-13249: -- Summary: Document webhdfs support for getting snapshottable directory list Key: HDFS-13249 URL: https://issues.apache.org/jira/browse/HDFS-13249 Project: Hadoop HDFS Issue Type: Bug Components: documentation, webhdfs Reporter: Lokesh Jain Assignee: Lokesh Jain This ticket is opened to document the WebHDFS: Add support for getting snasphottable directory list from HDFS-13141 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured
[
https://issues.apache.org/jira/browse/HDFS-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-12614:
--
Attachment: HDFS-12614.branch-2.patch
> FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider
> configured
> --
>
> Key: HDFS-12614
> URL: https://issues.apache.org/jira/browse/HDFS-12614
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HDFS-12614.01.patch, HDFS-12614.02.patch,
> HDFS-12614.03.patch, HDFS-12614.04.patch, HDFS-12614.branch-2.patch,
> HDFS-12614.test.01.patch
>
>
> When INodeAttributesProvider is configured, and when resolving path (like
> "/") and checking for permission, the following code when working on
> {{pathByNameArr}} throws NullPointerException.
> {noformat}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); <===
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {noformat}
> Looks like for paths like "/" where the split components based on delimiter
> "/" can be null, the pathByNameArr array can have null elements and can throw
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured
[
https://issues.apache.org/jira/browse/HDFS-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391680#comment-16391680
]
Kihwal Lee commented on HDFS-12614:
---
Attaching a branch-2 version. The only change is in the test. Lambda
expressions can't be used in branch-2 and older.
> FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider
> configured
> --
>
> Key: HDFS-12614
> URL: https://issues.apache.org/jira/browse/HDFS-12614
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HDFS-12614.01.patch, HDFS-12614.02.patch,
> HDFS-12614.03.patch, HDFS-12614.04.patch, HDFS-12614.branch-2.patch,
> HDFS-12614.test.01.patch
>
>
> When INodeAttributesProvider is configured, and when resolving path (like
> "/") and checking for permission, the following code when working on
> {{pathByNameArr}} throws NullPointerException.
> {noformat}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); <===
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {noformat}
> Looks like for paths like "/" where the split components based on delimiter
> "/" can be null, the pathByNameArr array can have null elements and can throw
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13190) Document WebHDFS support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391685#comment-16391685 ] Lokesh Jain commented on HDFS-13190: [~ajisakaa] Thanks for reviewing the patch! v2 patch addresses your comments. > Document WebHDFS support for snasphot diff > -- > > Key: HDFS-13190 > URL: https://issues.apache.org/jira/browse/HDFS-13190 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13190.001.patch, HDFS-13190.002.patch > > > This ticket is opened to document the WebHDFS: Add support for snasphot diff > from HDFS-13052 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13190) Document WebHDFS support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lokesh Jain updated HDFS-13190: --- Attachment: HDFS-13190.002.patch > Document WebHDFS support for snasphot diff > -- > > Key: HDFS-13190 > URL: https://issues.apache.org/jira/browse/HDFS-13190 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13190.001.patch, HDFS-13190.002.patch > > > This ticket is opened to document the WebHDFS: Add support for snasphot diff > from HDFS-13052 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated HDFS-13230:
---
Fix Version/s: 2.10.0
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13230) RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns
[
https://issues.apache.org/jira/browse/HDFS-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391695#comment-16391695
]
Íñigo Goiri commented on HDFS-13230:
{quote}
BTW, you forgot branch-2?
{quote}
Committed to branch-2 too. Thanks [~ywskycn] for the reminder... completely
forgot about it.
{quote}
BTW: I think the commit message is wrong: it should be HDFS-13230, not
HDFS-13232
{quote}
I messed up with the tabs while copying... let me see if I can amend the
message somewhat easily.
OK... I checked and this takes sorcery and it looks very risky, I'm going to
leave it as is.
If anybody is able to do it without issues, please feel free.
> RBF: ConnectionManager's cleanup task will compare each pool's own active
> conns with its total conns
>
>
> Key: HDFS-13230
> URL: https://issues.apache.org/jira/browse/HDFS-13230
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Chao Sun
>Priority: Minor
> Fix For: 3.1.0, 2.10.0, 2.9.1, 3.0.2, 3.2.0
>
> Attachments: HDFS-13230.000.patch, HDFS-13230.001.patch
>
>
> In the cleanup task:
> {code:java}
> long timeSinceLastActive = Time.now() - pool.getLastActiveTime();
> int total = pool.getNumConnections();
> int active = getNumActiveConnections();
> if (timeSinceLastActive > connectionCleanupPeriodMs ||
> {code}
> the 3rd line should be pool.getNumActiveConnections()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13186) [WRITE] Multipart Multinode uploader API + Implementations
[
https://issues.apache.org/jira/browse/HDFS-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391714#comment-16391714
]
genericqa commented on HDFS-13186:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 12m 54s{color}
| {color:red} root generated 2 new + 1294 unchanged - 0 fixed = 1296 total (was
1294) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 4s{color} | {color:orange} root: The patch generated 57 new + 26 unchanged
- 0 fixed = 83 total (was 26) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 1s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
40s{color} | {color:red} hadoop-common-project/hadoop-common generated 6 new +
0 unchanged - 0 fixed = 6 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
52s{color} | {color:red} hadoop-tools/hadoop-aws generated 5 new + 0 unchanged
- 0 fixed = 5 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
54s{color} | {color:red} hadoop-common-project_hadoop-common generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 45s{color}
| {color:red} hadoop-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
32s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
30s{color} | {color:green} hadoop-aws in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
34s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}120m 32s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-common-project/hadoop-common |
| | org.apache.hadoop.fs.BBPartHandle defines equals and uses
Object.hashCode() At BBPartHandle.java:Object.hashCode() At
BBPartHandle.java:[lines 43-47] |
| | org.apache.hadoop.fs.BBUploadHandle defines equals and uses
[jira] [Commented] (HDFS-13232) RBF: ConnectionPool should return first usable connection
[
https://issues.apache.org/jira/browse/HDFS-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391726#comment-16391726
]
Hudson commented on HDFS-13232:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13798 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13798/])
HDFS-13232. RBF: ConnectionManager's cleanup task will compare each (inigoiri:
rev 0c2b969e0161a068bf9ae013c4b95508dfb90a8a)
* (add)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestConnectionManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionPoolId.java
> RBF: ConnectionPool should return first usable connection
> -
>
> Key: HDFS-13232
> URL: https://issues.apache.org/jira/browse/HDFS-13232
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Ekanth S
>Priority: Minor
>
> In current ConnectionPool.getConnection(), it will return the first active
> connection:
> {code:java}
> for (int i=0; i int index = (threadIndex + i) % size;
> conn = tmpConnections.get(index);
> if (conn != null && !conn.isUsable()) {
> return conn;
> }
> }
> {code}
> Here "!conn.isUsable()" should be "conn.isUsable()".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391730#comment-16391730
]
Hanisha Koneru commented on HDFS-13239:
---
Thanks for working on this, [~bharatviswa].
Looks good to me overall, just few minor comments:
# We can directly assign the default policy to {{ecPolicyName}} and not need
another variable {{ecName}}.
# We would not need the below if condition as ecPolicyName cannot be null
anymore.
{code:java}
if (ecPolicyName == null){
System.out.println("Set default erasure coding policy " + ecName
+ " on " + path);
} {code}
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391751#comment-16391751
]
Bharat Viswanadham commented on HDFS-13239:
---
Hi [~hanishakoneru]
I have thought about it.
But, when the user has not specified the policy during set policy, to tell the
user, we are setting default Policy, I have done in this way. So, we can
specify in the message it is the default.
Otherwise, we will be changing the message printed to the user in default/and
user-specified cas the to same message.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13232) RBF: ConnectionPool should return first usable connection
[
https://issues.apache.org/jira/browse/HDFS-13232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391756#comment-16391756
]
Íñigo Goiri commented on HDFS-13232:
This is what happens when you mess up the commit message... Now Hudson thinks I
pushed to this one.
[~chris.douglas], some assist here, I committed HDFS-13230 but I messed up the
message and committed it as HDFS-13232, any action here?
> RBF: ConnectionPool should return first usable connection
> -
>
> Key: HDFS-13232
> URL: https://issues.apache.org/jira/browse/HDFS-13232
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Wei Yan
>Assignee: Ekanth S
>Priority: Minor
>
> In current ConnectionPool.getConnection(), it will return the first active
> connection:
> {code:java}
> for (int i=0; i int index = (threadIndex + i) % size;
> conn = tmpConnections.get(index);
> if (conn != null && !conn.isUsable()) {
> return conn;
> }
> }
> {code}
> Here "!conn.isUsable()" should be "conn.isUsable()".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391816#comment-16391816
]
Hanisha Koneru commented on HDFS-13239:
---
Thanks [~bharatviswa]. Got it now.
Can we have some boolean {{isDefault}} or something instead of {{ecName}}. The
two variables {{ecName}} and {{ecPolicyName}} are confusing :).
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13244) Add stack, conf, metrics links to utilities dropdown in NN webUI
[ https://issues.apache.org/jira/browse/HDFS-13244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391848#comment-16391848 ] Hanisha Koneru commented on HDFS-13244: --- +1 pending Jenkins. Will trigger a Jenkins run. > Add stack, conf, metrics links to utilities dropdown in NN webUI > > > Key: HDFS-13244 > URL: https://issues.apache.org/jira/browse/HDFS-13244 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13244.00.patch, Screen Shot 2018-03-07 at 11.28.27 > AM.png > > > Add stack, conf, metrics links to utilities dropdown in NN webUI > cc [~arpitagarwal] for suggesting this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured
[
https://issues.apache.org/jira/browse/HDFS-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391850#comment-16391850
]
Xiao Chen commented on HDFS-12614:
--
+1 based on diff. Thanks Kihwal!
> FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider
> configured
> --
>
> Key: HDFS-12614
> URL: https://issues.apache.org/jira/browse/HDFS-12614
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HDFS-12614.01.patch, HDFS-12614.02.patch,
> HDFS-12614.03.patch, HDFS-12614.04.patch, HDFS-12614.branch-2.patch,
> HDFS-12614.test.01.patch
>
>
> When INodeAttributesProvider is configured, and when resolving path (like
> "/") and checking for permission, the following code when working on
> {{pathByNameArr}} throws NullPointerException.
> {noformat}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); <===
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {noformat}
> Looks like for paths like "/" where the split components based on delimiter
> "/" can be null, the pathByNameArr array can have null elements and can throw
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13244) Add stack, conf, metrics links to utilities dropdown in NN webUI
[ https://issues.apache.org/jira/browse/HDFS-13244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391854#comment-16391854 ] Íñigo Goiri commented on HDFS-13244: Does Jenkins check anything for html and js? > Add stack, conf, metrics links to utilities dropdown in NN webUI > > > Key: HDFS-13244 > URL: https://issues.apache.org/jira/browse/HDFS-13244 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13244.00.patch, Screen Shot 2018-03-07 at 11.28.27 > AM.png > > > Add stack, conf, metrics links to utilities dropdown in NN webUI > cc [~arpitagarwal] for suggesting this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured
[
https://issues.apache.org/jira/browse/HDFS-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391888#comment-16391888
]
Kihwal Lee commented on HDFS-12614:
---
Committed to branch-2, branch-2.9 and branch-2.8.
> FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider
> configured
> --
>
> Key: HDFS-12614
> URL: https://issues.apache.org/jira/browse/HDFS-12614
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
> Fix For: 3.0.0, 2.10.0, 2.9.1, 2.8.4
>
> Attachments: HDFS-12614.01.patch, HDFS-12614.02.patch,
> HDFS-12614.03.patch, HDFS-12614.04.patch, HDFS-12614.branch-2.patch,
> HDFS-12614.test.01.patch
>
>
> When INodeAttributesProvider is configured, and when resolving path (like
> "/") and checking for permission, the following code when working on
> {{pathByNameArr}} throws NullPointerException.
> {noformat}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); <===
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {noformat}
> Looks like for paths like "/" where the split components based on delimiter
> "/" can be null, the pathByNameArr array can have null elements and can throw
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured
[
https://issues.apache.org/jira/browse/HDFS-12614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-12614:
--
Fix Version/s: 2.8.4
2.9.1
2.10.0
> FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider
> configured
> --
>
> Key: HDFS-12614
> URL: https://issues.apache.org/jira/browse/HDFS-12614
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
>Priority: Major
> Fix For: 3.0.0, 2.10.0, 2.9.1, 2.8.4
>
> Attachments: HDFS-12614.01.patch, HDFS-12614.02.patch,
> HDFS-12614.03.patch, HDFS-12614.04.patch, HDFS-12614.branch-2.patch,
> HDFS-12614.test.01.patch
>
>
> When INodeAttributesProvider is configured, and when resolving path (like
> "/") and checking for permission, the following code when working on
> {{pathByNameArr}} throws NullPointerException.
> {noformat}
> private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
> INode inode, int snapshotId) {
> INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
> if (getAttributesProvider() != null) {
> String[] elements = new String[pathIdx + 1];
> for (int i = 0; i < elements.length; i++) {
> elements[i] = DFSUtil.bytes2String(pathByNameArr[i]); <===
> }
> inodeAttrs = getAttributesProvider().getAttributes(elements,
> inodeAttrs);
> }
> return inodeAttrs;
> }
> {noformat}
> Looks like for paths like "/" where the split components based on delimiter
> "/" can be null, the pathByNameArr array can have null elements and can throw
> NPE.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13190) Document WebHDFS support for snasphot diff
[ https://issues.apache.org/jira/browse/HDFS-13190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391937#comment-16391937 ] Xiaoyu Yao commented on HDFS-13190: --- Thanks [~ljain] for the update. v2 patch LGTM, +1 pending Jenkins. > Document WebHDFS support for snasphot diff > -- > > Key: HDFS-13190 > URL: https://issues.apache.org/jira/browse/HDFS-13190 > Project: Hadoop HDFS > Issue Type: Bug > Components: documentation, webhdfs >Reporter: Xiaoyu Yao >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13190.001.patch, HDFS-13190.002.patch > > > This ticket is opened to document the WebHDFS: Add support for snasphot diff > from HDFS-13052 in WebHDFS.md. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13141) WebHDFS: Add support for getting snasphottable directory list
[ https://issues.apache.org/jira/browse/HDFS-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392068#comment-16392068 ] Xiaoyu Yao commented on HDFS-13141: --- Thanks [~ljain] for working on this. The patch looks good to me overall. I just have a few comments below: DFSOpsCountStatistics.java Line 91: Typo in op_get_snaphottable_directory_list should be op_get_snap*s*hottable_directory_list JsonUtilClient.java Line 776: I think we should set the flag based on the dirStatus instead of reset it to EnumSet.noneOf(HdfsFileStatus.Flags.class). After check HdfsFileStatus, it seems we don't have a getter for flags for HdfsFileStatus. Is this something we miss from HDFS-12681? cc: [~chris.douglas] TestWebHDFS.java Line 779: please add verification to ensure the dir flag returned from hdfs and webhdfs are the same. > WebHDFS: Add support for getting snasphottable directory list > - > > Key: HDFS-13141 > URL: https://issues.apache.org/jira/browse/HDFS-13141 > Project: Hadoop HDFS > Issue Type: Task > Components: webhdfs >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDFS-13141.001.patch, HDFS-13141.002.patch > > > This Jira aims to implement get snapshottable directory list operation for > webHdfs filesystem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392071#comment-16392071
]
Bharat Viswanadham commented on HDFS-13239:
---
[~hanishakoneru] Thanks for the review.
Addressed review comment in patch v01.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDFS-13239:
--
Attachment: HDFS-13239.01.patch
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13148) Unit test for EZ with KMS and Federation
[
https://issues.apache.org/jira/browse/HDFS-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392091#comment-16392091
]
Hanisha Koneru commented on HDFS-13148:
---
Hi [~shahrs87], thanks for the review.
If we make {{TestEncryptionZonesWithKMSandFederation}} extend
{{TestEncryptionZonesWithKMS}} then the former would run all the test cases in
the later with its initial {{setup}}. This would fail all the tests in
{{TestEncryptionZonesWithKMS}} run against the setup of
{{TestEncryptionZonesWithKMSandFederation}}. To get over this, we would have to
modify all the test cases in {{TestEncryptionZonesWithKMS}} and
{{TestEncryptionZones}} to work with the federated configuration setup. For
example, instead of using variable {{dfsAdmin}} in
{{TestEncryptionZonesWithKMS}}, we would have to change it to \{{dsfAdmin[0]}}
to match the federated setup.
I think it would just complicate all the three Tests by doing this. We could
instead have a \{{TestEncryptionZonesBaseTest}} class and make all the other
Tests extend this class. Or we could just keep the non-federated Tests and
federated Tests separate (as is in patch v03). Please let me know your thoughts.
> Unit test for EZ with KMS and Federation
>
>
> Key: HDFS-13148
> URL: https://issues.apache.org/jira/browse/HDFS-13148
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Hanisha Koneru
>Assignee: Hanisha Koneru
>Priority: Major
> Attachments: HDFS-13148.001.patch, HDFS-13148.002.patch,
> HDFS-13148.003.patch
>
>
> It would be good to have some unit tests for testing KMS and EZ on a
> federated cluster. We can start with basic EZ operations. For example, create
> EZs on two namespaces with different keys using one KMS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392096#comment-16392096
]
Hanisha Koneru commented on HDFS-13239:
---
Thanks [~bharatviswa].
+1 for patch v01 pending Jenkins.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13224: --- Summary: RBF: Resolvers to support mount points across multiple subclusters (was: RBF: Mount points across multiple subclusters) > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13136) Avoid taking FSN lock while doing group member lookup for FSD permission check
[ https://issues.apache.org/jira/browse/HDFS-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392106#comment-16392106 ] Xiaoyu Yao commented on HDFS-13136: --- Agree, HADOOP-11238 + HADOOP-13263 definitely will help the group lookup performance issue if configured properly. Together with this fix, even the slow warm-up period (where the cache does not even have an entry for certain user) won't holding FSN lock that could trigger a failover. > Avoid taking FSN lock while doing group member lookup for FSD permission check > -- > > Key: HDFS-13136 > URL: https://issues.apache.org/jira/browse/HDFS-13136 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Xiaoyu Yao >Assignee: Xiaoyu Yao >Priority: Major > Fix For: 3.1.0, 3.0.2, 3.2.0 > > Attachments: HDFS-13136-branch-3.0.001.patch, > HDFS-13136-branch-3.0.002.patch, HDFS-13136.001.patch, HDFS-13136.002.patch > > > Namenode has FSN lock and FSD lock. Most of the namenode operations need to > take FSN lock first and then FSD lock. The permission check is done via > FSPermissionChecker at FSD layer assuming FSN lock is taken. > The FSPermissionChecker constructor invokes callerUgi.getGroups() that can > take seconds sometimes. There are external cache scheme such SSSD and > internal cache scheme for group lookup. However, the delay could still occur > during cache refresh, which causes severe FSN lock contentions and > unresponsive namenode issues. > Checking the current code, we found that getBlockLocations(..) did it right > but some methods such as getFileInfo(..), getContentSummary(..) did it wrong. > This ticket is open to ensure the group lookup for permission checker is > outside the FSN lock. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392107#comment-16392107 ] Arpit Agarwal commented on HDFS-13056: -- Hi [~dennishuo], I haven't read your design doc in detail, but I have a question about variable length blocks. Supposing a 7 byte file is stored in multiple different schemes: # Single 7 byte block -> [7] # As two blocks -> [4, 3] # As 7 different blocks -> [1, 1, 1, 1, 1, 1, 1] Does this scheme ensure that the composite CRC will be the same in all cases? Or is there an assumption that all but the last block will be the same size? > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, > hdfs-file-composite-crc32-v1.pdf, hdfs-file-composite-crc32-v2.pdf, > hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13224: --- Attachment: HDFS-13224.003.patch > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch, HDFS-13224.003.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
Íñigo Goiri created HDFS-13250: -- Summary: RBF: Router to manage requests across multiple subclusters Key: HDFS-13250 URL: https://issues.apache.org/jira/browse/HDFS-13250 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Íñigo Goiri HDFS-131224 introduces the concept of mount points spanning multiple subclusters. The Router should distribute the requests across these subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392120#comment-16392120 ] Íñigo Goiri commented on HDFS-13224: In [^HDFS-13224.003.patch] I left only the resolvers for multiple subclusters. I created HDFS-13250 to add the support to the Router itself. > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch, HDFS-13224.003.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned HDFS-13250: -- Assignee: Íñigo Goiri > RBF: Router to manage requests across multiple subclusters > -- > > Key: HDFS-13250 > URL: https://issues.apache.org/jira/browse/HDFS-13250 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HDFS-131224 introduces the concept of mount points spanning multiple > subclusters. The Router should distribute the requests across these > subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11394) Add method for getting erasure coding policy through WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392122#comment-16392122
]
Arpit Agarwal commented on HDFS-11394:
--
The patch looks good.
+1 with the javac warnings addressed. They can be fixed by adding
{{StandardCharsets.UTF_8}} as the second parameter to {{IOUtils#toString}}.
> Add method for getting erasure coding policy through WebHDFS
> -
>
> Key: HDFS-11394
> URL: https://issues.apache.org/jira/browse/HDFS-11394
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: erasure-coding, namenode
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Major
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-11394.005.patch, HDFS-11394.01.patch,
> HDFS-11394.02.patch, HDFS-11394.03.patch, HDFS-11394.04.patch
>
>
> We can expose erasure coding policy by erasure coded directory through
> WebHDFS method as well as storage policy. This information can be used by
> NameNode Web UI and show the detail of erasure coded directories.
> see: HDFS-8196
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13250: --- Description: HDFS-13124 introduces the concept of mount points spanning multiple subclusters. The Router should distribute the requests across these subclusters. (was: HDFS-131224 introduces the concept of mount points spanning multiple subclusters. The Router should distribute the requests across these subclusters.) > RBF: Router to manage requests across multiple subclusters > -- > > Key: HDFS-13250 > URL: https://issues.apache.org/jira/browse/HDFS-13250 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HDFS-13124 introduces the concept of mount points spanning multiple > subclusters. The Router should distribute the requests across these > subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13250: --- Attachment: HDFS-13250.000.patch > RBF: Router to manage requests across multiple subclusters > -- > > Key: HDFS-13250 > URL: https://issues.apache.org/jira/browse/HDFS-13250 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13250.000.patch > > > HDFS-13124 introduces the concept of mount points spanning multiple > subclusters. The Router should distribute the requests across these > subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13250) RBF: Router to manage requests across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392136#comment-16392136 ] Íñigo Goiri commented on HDFS-13250: I added [^HDFS-13250.000.patch] temporarily until HDFS-13224 gets committed. As [~linyiqun] mentioned there, we need to go over all the methods and decide which ones are executed in all mount points. > RBF: Router to manage requests across multiple subclusters > -- > > Key: HDFS-13250 > URL: https://issues.apache.org/jira/browse/HDFS-13250 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13250.000.patch > > > HDFS-13124 introduces the concept of mount points spanning multiple > subclusters. The Router should distribute the requests across these > subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392142#comment-16392142
]
Xiao Chen commented on HDFS-13239:
--
Thanks [~hanishakoneru] for reporting the issue and [~bharatviswa] for fixing
it.
I don't think the fix is correct though. The configuration is a NN property, so
even though we can read it in ECAdmin, there is no way we can guarantee its
value is the same as the NN.
Instead of printing out the default policy name, how about we just literally
say 'the default erasure coding policy' in the message?
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[
https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392165#comment-16392165
]
genericqa commented on HDFS-13224:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
40s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m
0s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13224 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913673/HDFS-13224.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 15e5c4c58782 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 113f401 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/23358/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDFS-Build/23358/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/23358/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/23358/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392175#comment-16392175
]
Bharat Viswanadham commented on HDFS-13239:
---
Hi [~xiaochen]
As we are reading the value available from the config in ECAdmin, and we are
passing the same to dfs.setErasureCodingPolicy(p, ecPolicyName);
That is the value we are printing as default. So, what is the default policy
set in the client(where command is invoked), that is the default policy with
which ecPolicy will be set for the path.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13056) Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts
[ https://issues.apache.org/jira/browse/HDFS-13056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392176#comment-16392176 ] Dennis Huo commented on HDFS-13056: --- Variable-sized blocks within a file are indeed supported (though in your examples there are still uniform blocksizes within a file except for last file); in addition to your three examples, the following would also return the same value as another example: # [1, 3, 2, 1] This is implemented by FileChecksumHelper always using block.getBlockSize() as the data length for each block rather than assuming what the block size should be, except for the last block, where I don't trust block.getBlockSize(). In the last block, it turned out that some flows modified the reported block-size in-place and others didn't so I just made sure to manually compute the correct number of bytes associated with the last block. There are constraints on chunk and cell sizes, but these are based on inherent constraints in the system rather than assumptions in my algorithm; cell size must be uniform within a block group (but can vary in different block groups within a single file), which is already a fundamental assumption in striped reader anyways, and chunk size must be uniform within a block (which is also fundamental because block metadata doesn't have any way to indicate changes of bytes-per-crc midway through the block). > Expose file-level composite CRCs in HDFS which are comparable across > different instances/layouts > > > Key: HDFS-13056 > URL: https://issues.apache.org/jira/browse/HDFS-13056 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, distcp, erasure-coding, federation, hdfs >Affects Versions: 3.0.0 >Reporter: Dennis Huo >Assignee: Dennis Huo >Priority: Major > Attachments: HDFS-13056-branch-2.8.001.patch, > HDFS-13056-branch-2.8.002.patch, HDFS-13056-branch-2.8.003.patch, > HDFS-13056-branch-2.8.004.patch, HDFS-13056-branch-2.8.005.patch, > HDFS-13056-branch-2.8.poc1.patch, HDFS-13056.001.patch, HDFS-13056.002.patch, > HDFS-13056.003.patch, HDFS-13056.003.patch, HDFS-13056.004.patch, > HDFS-13056.005.patch, HDFS-13056.006.patch, HDFS-13056.007.patch, > HDFS-13056.008.patch, Reference_only_zhen_PPOC_hadoop2.6.X.diff, > hdfs-file-composite-crc32-v1.pdf, hdfs-file-composite-crc32-v2.pdf, > hdfs-file-composite-crc32-v3.pdf > > > FileChecksum was first introduced in > [https://issues-test.apache.org/jira/browse/HADOOP-3981] and ever since then > has remained defined as MD5-of-MD5-of-CRC, where per-512-byte chunk CRCs are > already stored as part of datanode metadata, and the MD5 approach is used to > compute an aggregate value in a distributed manner, with individual datanodes > computing the MD5-of-CRCs per-block in parallel, and the HDFS client > computing the second-level MD5. > > A shortcoming of this approach which is often brought up is the fact that > this FileChecksum is sensitive to the internal block-size and chunk-size > configuration, and thus different HDFS files with different block/chunk > settings cannot be compared. More commonly, one might have different HDFS > clusters which use different block sizes, in which case any data migration > won't be able to use the FileChecksum for distcp's rsync functionality or for > verifying end-to-end data integrity (on top of low-level data integrity > checks applied at data transfer time). > > This was also revisited in https://issues.apache.org/jira/browse/HDFS-8430 > during the addition of checksum support for striped erasure-coded files; > while there was some discussion of using CRC composability, it still > ultimately settled on hierarchical MD5 approach, which also adds the problem > that checksums of basic replicated files are not comparable to striped files. > > This feature proposes to add a "COMPOSITE-CRC" FileChecksum type which uses > CRC composition to remain completely chunk/block agnostic, and allows > comparison between striped vs replicated files, between different HDFS > instances, and possible even between HDFS and other external storage systems. > This feature can also be added in-place to be compatible with existing block > metadata, and doesn't need to change the normal path of chunk verification, > so is minimally invasive. This also means even large preexisting HDFS > deployments could adopt this feature to retroactively sync data. A detailed > design document can be found here: > https://storage.googleapis.com/dennishuo/hdfs-file-composite-crc32-v1.pdf -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues
[jira] [Commented] (HDFS-13244) Add stack, conf, metrics links to utilities dropdown in NN webUI
[ https://issues.apache.org/jira/browse/HDFS-13244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392179#comment-16392179 ] Hanisha Koneru commented on HDFS-13244: --- No but just wanted Jenkins to +1 it to follow conventions :) > Add stack, conf, metrics links to utilities dropdown in NN webUI > > > Key: HDFS-13244 > URL: https://issues.apache.org/jira/browse/HDFS-13244 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13244.00.patch, Screen Shot 2018-03-07 at 11.28.27 > AM.png > > > Add stack, conf, metrics links to utilities dropdown in NN webUI > cc [~arpitagarwal] for suggesting this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392175#comment-16392175
]
Bharat Viswanadham edited comment on HDFS-13239 at 3/9/18 12:39 AM:
Hi [~xiaochen]
As we are reading the value available from the config in ECAdmin, and we are
passing the same to dfs.setErasureCodingPolicy(p, ecPolicyName);
That is the value we are printing as default. So, what is the default policy
set in the client(where command is invoked), that is the default policy with
which ecPolicy will be set for the path.
was (Author: bharatviswa):
Hi [~xiaochen]
As we are reading the value available from the config in ECAdmin, and we are
passing the same to dfs.setErasureCodingPolicy(p, ecPolicyName);
That is the value we are printing as default. So, what is the default policy
set in the client(where command is invoked), that is the default policy with
which ecPolicy will be set for the path.
> Fix non-empty dir warning message when setting default EC policy
>
>
> Key: HDFS-13239
> URL: https://issues.apache.org/jira/browse/HDFS-13239
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hanisha Koneru
>Assignee: Bharat Viswanadham
>Priority: Minor
> Attachments: HDFS-13239.00.patch, HDFS-13239.01.patch
>
>
> When EC policy is set on a non-empty directory, the following warning message
> is given:
> {code}
> $hdfs ec -setPolicy -policy RS-6-3-1024k -path /ec1
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to RS-6-3-1024k
> {code}
> When we do not specify the -policy parameter when setting EC policy on a
> directory, it takes the default EC policy. Setting default EC policy in this
> way on a non-empty directory gives the following warning message:
> {code}
> $hdfs ec -setPolicy -path /ec2
> Warning: setting erasure coding policy on a non-empty directory will not
> automatically convert existing files to null
> {code}
> Notice that the warning message in the 2nd case has the ecPolicy name shown
> as null. We should instead give the default EC policy name in this message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options
[ https://issues.apache.org/jira/browse/HDFS-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392192#comment-16392192 ] Arpit Agarwal commented on HDFS-13235: -- Thanks for updating the docs [~bharatviswa]. Minor comments: # _Or provide complete number in seconds (such as 30 for 30 seconds)._ This sentence should be removed, since we later mention that the default is milliseconds. # grammar nitpick: Change _Skips date check_ to _Skip date check_. # grammar nitpick: Change _Can use the following suffix_ to _Supports the following suffixes_ > DiskBalancer: Update Documentation to add newly added options > - > > Key: HDFS-13235 > URL: https://issues.apache.org/jira/browse/HDFS-13235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13235.00.patch > > > HDFS-13181 added dfs.disk.balancer.plan.valid.interval > HDFS-13178 added skipDateCheck Option -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13224) RBF: Resolvers to support mount points across multiple subclusters
[ https://issues.apache.org/jira/browse/HDFS-13224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-13224: --- Attachment: HDFS-13224.004.patch > RBF: Resolvers to support mount points across multiple subclusters > -- > > Key: HDFS-13224 > URL: https://issues.apache.org/jira/browse/HDFS-13224 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-13224.000.patch, HDFS-13224.001.patch, > HDFS-13224.002.patch, HDFS-13224.003.patch, HDFS-13224.004.patch > > > Currently, a mount point points to a single subcluster. We should be able to > spread files in a mount point across subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options
[ https://issues.apache.org/jira/browse/HDFS-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDFS-13235: -- Attachment: HDFS-13235.01.patch > DiskBalancer: Update Documentation to add newly added options > - > > Key: HDFS-13235 > URL: https://issues.apache.org/jira/browse/HDFS-13235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13235.00.patch, HDFS-13235.01.patch > > > HDFS-13181 added dfs.disk.balancer.plan.valid.interval > HDFS-13178 added skipDateCheck Option -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options
[ https://issues.apache.org/jira/browse/HDFS-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392218#comment-16392218 ] Bharat Viswanadham commented on HDFS-13235: --- Thank You [~arpitagarwal] for review. Addressed review comments in v01 patch. > DiskBalancer: Update Documentation to add newly added options > - > > Key: HDFS-13235 > URL: https://issues.apache.org/jira/browse/HDFS-13235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13235.00.patch, HDFS-13235.01.patch > > > HDFS-13181 added dfs.disk.balancer.plan.valid.interval > HDFS-13178 added skipDateCheck Option -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11394) Add method for getting erasure coding policy through WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392240#comment-16392240 ] Hanisha Koneru commented on HDFS-11394: --- Thanks for the review, [~arpitagarwal]. Addressed javadoc and checkstyle issues in patch v06. > Add method for getting erasure coding policy through WebHDFS > - > > Key: HDFS-11394 > URL: https://issues.apache.org/jira/browse/HDFS-11394 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, namenode >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11394.005.patch, HDFS-11394.006.patch, > HDFS-11394.01.patch, HDFS-11394.02.patch, HDFS-11394.03.patch, > HDFS-11394.04.patch > > > We can expose erasure coding policy by erasure coded directory through > WebHDFS method as well as storage policy. This information can be used by > NameNode Web UI and show the detail of erasure coded directories. > see: HDFS-8196 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11394) Add method for getting erasure coding policy through WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-11394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDFS-11394: -- Attachment: HDFS-11394.006.patch > Add method for getting erasure coding policy through WebHDFS > - > > Key: HDFS-11394 > URL: https://issues.apache.org/jira/browse/HDFS-11394 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding, namenode >Reporter: Kai Sasaki >Assignee: Kai Sasaki >Priority: Major > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-11394.005.patch, HDFS-11394.006.patch, > HDFS-11394.01.patch, HDFS-11394.02.patch, HDFS-11394.03.patch, > HDFS-11394.04.patch > > > We can expose erasure coding policy by erasure coded directory through > WebHDFS method as well as storage policy. This information can be used by > NameNode Web UI and show the detail of erasure coded directories. > see: HDFS-8196 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13235) DiskBalancer: Update Documentation to add newly added options
[ https://issues.apache.org/jira/browse/HDFS-13235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392260#comment-16392260 ] Arpit Agarwal commented on HDFS-13235: -- +1 pending Jenkins. > DiskBalancer: Update Documentation to add newly added options > - > > Key: HDFS-13235 > URL: https://issues.apache.org/jira/browse/HDFS-13235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13235.00.patch, HDFS-13235.01.patch > > > HDFS-13181 added dfs.disk.balancer.plan.valid.interval > HDFS-13178 added skipDateCheck Option -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13239) Fix non-empty dir warning message when setting default EC policy
[
https://issues.apache.org/jira/browse/HDFS-13239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392270#comment-16392270
]
genericqa commented on HDFS-13239:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 54s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}140m 36s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}189m 1s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | HDFS-13239 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12913667/HDFS-13239.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux 14cdede09291 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 113f401 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23357/artifact/out/patch-unit-hadoop-hdfs
[jira] [Created] (HDFS-13251) Avoid using hard coded datanode data dirs in unit tests
Xiaoyu Yao created HDFS-13251:
-
Summary: Avoid using hard coded datanode data dirs in unit tests
Key: HDFS-13251
URL: https://issues.apache.org/jira/browse/HDFS-13251
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Xiaoyu Yao
Assignee: Ajay Kumar
There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
names.
* TestDataNodeVolumeFailureToleration
* TestDataNodeVolumeFailureReporting
* TestDiskBalancerCommand
* TestBlockStatsMXBean
* TestDataNodeVolumeMetrics
* TestDFSAdmin
* TestDataNodeHotSwapVolumes
* TestDataNodeVolumeFailure
This ticket is opened to use
{code} MiniDFSCluster#getInstanceStorageDir() \{code} instead of
new File(cluster.getDataDirectory(), "data1");
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13251) Avoid using hard coded datanode data dirs in unit tests
[
https://issues.apache.org/jira/browse/HDFS-13251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-13251:
--
Description:
There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
names.
* TestDataNodeVolumeFailureToleration
* TestDataNodeVolumeFailureReporting
* TestDiskBalancerCommand
* TestBlockStatsMXBean
* TestDataNodeVolumeMetrics
* TestDFSAdmin
* TestDataNodeHotSwapVolumes
* TestDataNodeVolumeFailure
This ticket is opened to use
{code:java}
MiniDFSCluster#getInstanceStorageDir(0, 1);
instead of like below
new File(cluster.getDataDirectory(), "data1");{code}
was:
There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
names.
* TestDataNodeVolumeFailureToleration
* TestDataNodeVolumeFailureReporting
* TestDiskBalancerCommand
* TestBlockStatsMXBean
* TestDataNodeVolumeMetrics
* TestDFSAdmin
* TestDataNodeHotSwapVolumes
* TestDataNodeVolumeFailure
This ticket is opened to use
{code} MiniDFSCluster#getInstanceStorageDir() \{code} instead of
new File(cluster.getDataDirectory(), "data1");
> Avoid using hard coded datanode data dirs in unit tests
> ---
>
> Key: HDFS-13251
> URL: https://issues.apache.org/jira/browse/HDFS-13251
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Xiaoyu Yao
>Assignee: Ajay Kumar
>Priority: Major
>
> There are a few unit tests that rely on hard-coded MiniDFSCluster data dir
> names.
>
> * TestDataNodeVolumeFailureToleration
> * TestDataNodeVolumeFailureReporting
> * TestDiskBalancerCommand
> * TestBlockStatsMXBean
> * TestDataNodeVolumeMetrics
> * TestDFSAdmin
> * TestDataNodeHotSwapVolumes
> * TestDataNodeVolumeFailure
> This ticket is opened to use
> {code:java}
> MiniDFSCluster#getInstanceStorageDir(0, 1);
> instead of like below
> new File(cluster.getDataDirectory(), "data1");{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
