[jira] [Commented] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180315#comment-17180315
]
Hadoop QA commented on HDFS-15533:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 3m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
23s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 18m
18s{color} | {color:red} root in trunk failed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 19m
48s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
18s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
31s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
45s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m
47s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 21m 47s{color}
| {color:red} root-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 223 new + 1833 unchanged
- 9 fixed = 2056 total (was 1842) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m
22s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 20m 22s{color}
| {color:red} root-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 with
JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 12 new +
1939 unchanged - 8 fixed = 1951 total (was 1947) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 11s{color} | {color:orange} root: The patch generated 7 new + 242 unchanged
- 0 fixed = 249 total (was 242) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-03.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: (was: HDFS-15535-03.patch)
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-03.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180327#comment-17180327
]
Ayush Saxena commented on HDFS-15535:
-
Thanx [~elgoiri] for the review. I made changes in v3, to return a {{Map}}
which contains the invokedLocation. Give a check if this makes more sense.
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180360#comment-17180360
]
Hadoop QA commented on HDFS-15535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
50s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
12s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 2 new +
0 unchanged - 0 fixed = 2 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
26s{color} | {color:green} hadoop-hdfs-rbf in the patch
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-04.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17179470#comment-17179470 ] liusheng edited comment on HDFS-15098 at 8/19/20, 8:54 AM: --- Hi [~lindongdong], Thank you for your review, I am not an expert and not sure about the potential compatible issue in rolling upgrade process, but should we keep consistency in interacted components to avoid old jars called by new native libaries when do rolling upgrade ? I have tried adding the method definition you mentioned, but CI raised error as bellow: !image-2020-08-19-16-54-41-341.png! Thank for your suggestion about [OpensslSecureRandom.c|https://github.com/apache/hadoop/pull/2211/files#diff-3ee504e8c2a27c840c39c4496a27cc02], will check later. was (Author: seanlau): Hi [~lindongdong], Thank you for your review, I am not an expert and not sure about the potential compatible issue in rolling upgrade process, but should we keep consistency in interacted components to avoid old jars called by new native libaries when do rolling upgrade ? I have tried adding the method definition you mentioned, but CI raised error as bellow: !image-2020-08-18-16-40-30-090.png! Thank for your suggestion about [OpensslSecureRandom.c|https://github.com/apache/hadoop/pull/2211/files#diff-3ee504e8c2a27c840c39c4496a27cc02], will check later. > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: liusheng >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch, image-2020-08-19-16-54-41-341.png > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15098) Add SM4 encryption method for HDFS
[ https://issues.apache.org/jira/browse/HDFS-15098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180372#comment-17180372 ] liusheng commented on HDFS-15098: - Hi [~lindongdong], sorry, I have update the pic again. > Add SM4 encryption method for HDFS > -- > > Key: HDFS-15098 > URL: https://issues.apache.org/jira/browse/HDFS-15098 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 3.4.0 >Reporter: liusheng >Assignee: liusheng >Priority: Major > Labels: sm4 > Attachments: HDFS-15098.001.patch, HDFS-15098.002.patch, > HDFS-15098.003.patch, HDFS-15098.004.patch, HDFS-15098.005.patch, > HDFS-15098.006.patch, HDFS-15098.007.patch, HDFS-15098.008.patch, > HDFS-15098.009.patch, image-2020-08-19-16-54-41-341.png > > > SM4 (formerly SMS4)is a block cipher used in the Chinese National Standard > for Wireless LAN WAPI (Wired Authentication and Privacy Infrastructure). > SM4 was a cipher proposed to for the IEEE 802.11i standard, but has so far > been rejected by ISO. One of the reasons for the rejection has been > opposition to the WAPI fast-track proposal by the IEEE. please see: > [https://en.wikipedia.org/wiki/SM4_(cipher)] > > *Use sm4 on hdfs as follows:* > 1.Configure Hadoop KMS > 2.test HDFS sm4 > hadoop key create key1 -cipher 'SM4/CTR/NoPadding' > hdfs dfs -mkdir /benchmarks > hdfs crypto -createZone -keyName key1 -path /benchmarks > *requires:* > 1.openssl version >=1.1.1 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-05.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180436#comment-17180436
]
Hadoop QA commented on HDFS-15535:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
28s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
48s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF Lic
[jira] [Updated] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tomscut updated HDFS-15380:
---
Priority: Major (was: Minor)
> RBF: Could not fetch real remote IP in RouterWebHdfsMethods
> ---
>
> Key: HDFS-15380
> URL: https://issues.apache.org/jira/browse/HDFS-15380
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Major
> Labels: router, webhdfs
> Attachments: HDFS-15380.001.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> We plan to add audit log for hdfs router, then we fetch remote ip via
> Server.getRemoteIp(), but the result is "localhost/127.0.0.1".
>
> "REMOTE_ADDRESS" in RouterWebHdfsMethods.java is a ThreadLocal field,
> setting in construction method RouterWebHdfsMethods() and init(). When we
> call method Server.getRemoteIp() to fetch remote ip, the thread would be
> changed, so the ThreadLocal field "REMOTE_ADDRESS" is null, and would be
> passed to "localhost/127.0.0.1" via InetAddress.getByName().
>
> So we can change this field "REMOTE_ADDRESS" to a String value, just like
> NamenodeWebHdfsMethods does.
>
> I printed thread name and the value of "REMOTE_ADDRESS" in log, the log is
> shown below:
> {code:java}
> 2020-05-27 19:15:18,797 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:(138)) - RouterWebHdfsMethods
> REMOTE_ADDRESS: 14.39.39.28, current thread: qtp476579021-1090
> 2020-05-27 19:15:18,827 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:init(150)) - init REMOTE_ADDRESS: 14.39.39.28,
> current thread: qtp476579021-1090
> 2020-05-27 19:15:18,836 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,837 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:getRemoteAddr(170)) - getRemoteAddr
> REMOTE_ADDRESS: null, current thread: IPC Server handler 75 on
> 2020-05-27 19:15:18,883 INFO router.RouterWebHdfsMethods
> (RouterWebHdfsMethods.java:reset(164)) - reset REMOTE_ADDRESS: null, current
> thread: IPC Server handler 75 on
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180465#comment-17180465
]
Hadoop QA commented on HDFS-15535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
27s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 51s{color}
| {color:red} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color}
[jira] [Commented] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180482#comment-17180482 ] Fei Hui commented on HDFS-15422: [~kihwal] Thanks for reporting and the fix. Can we push this fix to trunk? > Reported IBR is partially replaced with stored info when queuing. > - > > Key: HDFS-15422 > URL: https://issues.apache.org/jira/browse/HDFS-15422 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Kihwal Lee >Priority: Critical > > When queueing an IBR (incremental block report) on a standby namenode, some > of the reported information is being replaced with the existing stored > information. This can lead to false block corruption. > We had a namenode, after transitioning to active, started reporting missing > blocks with "SIZE_MISMATCH" as corrupt reason. These were blocks that were > appended and the sizes were actually correct on the datanodes. Upon further > investigation, it was determined that the namenode was queueing IBRs with > altered information. > Although it sounds bad, I am not making it blocker -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
[
https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180485#comment-17180485
]
Fei Hui commented on HDFS-14852:
[~sodonnell] [~kihwal] Can we move forward and fix this issue?
> Remove of LowRedundancyBlocks do NOT remove the block from all queues
> -
>
> Key: HDFS-14852
> URL: https://issues.apache.org/jira/browse/HDFS-14852
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: CorruptBlocksMismatch.png, HDFS-14852.001.patch,
> HDFS-14852.002.patch, HDFS-14852.003.patch, HDFS-14852.004.patch,
> HDFS-14852.005.patch, screenshot-1.png
>
>
> LowRedundancyBlocks.java
> {code:java}
> // Some comments here
> if(priLevel >= 0 && priLevel < LEVEL
> && priorityQueues.get(priLevel).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
> + " from priority queue {}",
> block, priLevel);
> decrementBlockStat(block, priLevel, oldExpectedReplicas);
> return true;
> } else {
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> for (int i = 0; i < LEVEL; i++) {
> if (i != priLevel && priorityQueues.get(i).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
> " {} from priority queue {}", block, i);
> decrementBlockStat(block, i, oldExpectedReplicas);
> return true;
> }
> }
> }
> return false;
> }
> {code}
> Source code is above, the comments as follow
> {quote}
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> {quote}
> The function "remove" does NOT remove the block from all queues.
> Function add from LowRedundancyBlocks.java is used on some places and maybe
> one block in two or more queues.
> We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is
> related to this.
> Upload initial patch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15380) RBF: Could not fetch real remote IP in RouterWebHdfsMethods
[
https://issues.apache.org/jira/browse/HDFS-15380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180502#comment-17180502
]
Hadoop QA commented on HDFS-15380:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
19s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 28s{color}
| {color:red} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color}
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180526#comment-17180526
]
Ayush Saxena commented on HDFS-15535:
-
In v5 instead of a Map as return type, have added a class, can check which
approach seems better, v4 with Map as return type or v5
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
[
https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180604#comment-17180604
]
Hadoop QA commented on HDFS-14852:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
37s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 9s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 45s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
[jira] [Commented] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[ https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180615#comment-17180615 ] Hemanth Boyina commented on HDFS-15510: --- thanks for the comments [~elgoiri] updated the patch , please review > RBF: Quota and Content Summary was not correct in Multiple Destinations > --- > > Key: HDFS-15510 > URL: https://issues.apache.org/jira/browse/HDFS-15510 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Critical > Attachments: 15510.png, HDFS-15510.001.patch, HDFS-15510.002.patch > > > steps : > *) create a mount entry with multiple destinations ( for suppose 2) > *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content > Summary on the Mount Entry shows NS quota as 20 > *) Create 10 files through router, on creating 11th file , NS Quota Exceeded > Exception is coming > though the Content Summary showing the NS quota as 20 , we are not able to > create 20 files > > the problem here is router stores the mount entry's NS quota as 10 , but > invokes NS quota on both the name services by set NS quota as 10 , so content > summary on mount entry aggregates the content summary of both the name > services by making NS quota as 20 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[ https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hemanth Boyina updated HDFS-15510: -- Attachment: HDFS-15510.002.patch > RBF: Quota and Content Summary was not correct in Multiple Destinations > --- > > Key: HDFS-15510 > URL: https://issues.apache.org/jira/browse/HDFS-15510 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Hemanth Boyina >Assignee: Hemanth Boyina >Priority: Critical > Attachments: 15510.png, HDFS-15510.001.patch, HDFS-15510.002.patch > > > steps : > *) create a mount entry with multiple destinations ( for suppose 2) > *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content > Summary on the Mount Entry shows NS quota as 20 > *) Create 10 files through router, on creating 11th file , NS Quota Exceeded > Exception is coming > though the Content Summary showing the NS quota as 20 , we are not able to > create 20 files > > the problem here is router stores the mount entry's NS quota as 10 , but > invokes NS quota on both the name services by set NS quota as 10 , so content > summary on mount entry aggregates the content summary of both the name > services by making NS quota as 20 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
[
https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180621#comment-17180621
]
Stephen O'Donnell commented on HDFS-14852:
--
I just re-checked some of the older comments here and I think my suggestion in
this comment is a good way forward, as it would maintain the existing
performance and should fix the issue:
https://issues.apache.org/jira/browse/HDFS-14852?focusedCommentId=16992362&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16992362
> Remove of LowRedundancyBlocks do NOT remove the block from all queues
> -
>
> Key: HDFS-14852
> URL: https://issues.apache.org/jira/browse/HDFS-14852
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: CorruptBlocksMismatch.png, HDFS-14852.001.patch,
> HDFS-14852.002.patch, HDFS-14852.003.patch, HDFS-14852.004.patch,
> HDFS-14852.005.patch, screenshot-1.png
>
>
> LowRedundancyBlocks.java
> {code:java}
> // Some comments here
> if(priLevel >= 0 && priLevel < LEVEL
> && priorityQueues.get(priLevel).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
> + " from priority queue {}",
> block, priLevel);
> decrementBlockStat(block, priLevel, oldExpectedReplicas);
> return true;
> } else {
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> for (int i = 0; i < LEVEL; i++) {
> if (i != priLevel && priorityQueues.get(i).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
> " {} from priority queue {}", block, i);
> decrementBlockStat(block, i, oldExpectedReplicas);
> return true;
> }
> }
> }
> return false;
> }
> {code}
> Source code is above, the comments as follow
> {quote}
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> {quote}
> The function "remove" does NOT remove the block from all queues.
> Function add from LowRedundancyBlocks.java is used on some places and maybe
> one block in two or more queues.
> We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is
> related to this.
> Upload initial patch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[
https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180637#comment-17180637
]
Ayush Saxena commented on HDFS-15510:
-
Was looking after the quota stuff in case of {{Multiple Destinations}} and on a
quick glance the approach #3 actually made sense in the first go. But, Since
there was even #2 suggested above by [~elgoiri], I tried digging a bit more.
I think the approach #3 does good when quota is set on the mount entry as
{{/router_test}}, but if this directory follows {{isPathAll}} order and you
create a directory {{/router_test/dir}} and then set quota on it, If there are
two namespaces, the {{setQuota}} will be a concurrent call and set on both
namespaces, The point is with #3 in count command the quota will shows correct,
but will it throw Quota exceeded exception at the value? I tried a quick UT and
it didn't. since for the namespace directories the quota is maintained at
router, For that case something near to approach #2, makes the functionality
work(Not a best solution either, due to restriction on randomness of
distribution), Can you check once, if for directories quota is protected, not
just {{mount entries}}
if that is protected, then coming to current patch, I tried this as well for
the mount entries, The present code, overlooks the overflow of {{LONG}} due to
addition of quota, set nsQuota to {{LONG.MAX-2}} in the current code and this
goes wrong. So, that also I guess needs to taken care of.
This the modified UT for the directories stuff :
{code:java}
@Test
public void testContentSummaryWithMultipleDest() throws Exception {
MountTable addEntry;
long nsQuota = 5;
long ssQuota = 100;
Path path = new Path("/router_test");
Map destMap = new HashMap<>();
destMap.put("ns0", "/router_test");
destMap.put("ns1", "/router_test");
nnFs0.mkdirs(path);
nnFs1.mkdirs(path);
addEntry = MountTable.newInstance("/router_test", destMap);
addEntry.setDestOrder(DestinationOrder.HASH_ALL);
assertTrue(addMountTable(addEntry));
RouterQuotaUpdateService updateService =
routerContext.getRouter().getQuotaCacheUpdateService();
updateService.periodicInvoke();
routerFs.mkdirs(new Path("/router_test/dir"));
routerFs.setQuota(new Path("/router_test/dir"),nsQuota,ssQuota);
for(int i=0; i RBF: Quota and Content Summary was not correct in Multiple Destinations
> ---
>
> Key: HDFS-15510
> URL: https://issues.apache.org/jira/browse/HDFS-15510
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Hemanth Boyina
>Assignee: Hemanth Boyina
>Priority: Critical
> Attachments: 15510.png, HDFS-15510.001.patch, HDFS-15510.002.patch
>
>
> steps :
> *) create a mount entry with multiple destinations ( for suppose 2)
> *) Set NS quota as 10 for mount entry by dfsrouteradmin command, Content
> Summary on the Mount Entry shows NS quota as 20
> *) Create 10 files through router, on creating 11th file , NS Quota Exceeded
> Exception is coming
> though the Content Summary showing the NS quota as 20 , we are not able to
> create 20 files
>
> the problem here is router stores the mount entry's NS quota as 10 , but
> invokes NS quota on both the name services by set NS quota as 10 , so content
> summary on mount entry aggregates the content summary of both the name
> services by making NS quota as 20
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Description:
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I have a patch and will post it in some time.
was:
I have been working on a thought from last week is that, we wanted to provide
DFS compatible APIs with mount functionality. So, that existing DFS
applications can work with out class cast issues.
When we tested with other components like Hive and HBase, I noticed some
classcast issues.
{code:java}
HBase example:
java.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
org.apache.hadoop.hdfs.DistributedFileSystem at
org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
at java.lang.Thread.run(Thread.java:748){code}
{code:java}
Hive:
|io.AcidUtils|: Failed to get files with ID; using regular API: Only supported
for DFS; got class
org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
SO, here the implementation details are like follows:
We extended DistributedFileSystem and created a class called "
ViewDistributedFileSystem"
This vfs=ViewFirstibutedFileSystem, try to initialize
ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
to initialize due to no mount points, or other errors, it will just fallback to
regular DFS init. If users does not configure any mount, system will behave
exactly like today's DFS. If there are mount points, vfs functionality will
come under DFS.
I will a patch and will post it in some time.
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme canno
[jira] [Resolved] (HDFS-15533) Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G resolved HDFS-15533.
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
I have just committed this to trunk. Thanks [~ayushtkn] for reviews!
> Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside
> -
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15510) RBF: Quota and Content Summary was not correct in Multiple Destinations
[
https://issues.apache.org/jira/browse/HDFS-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180674#comment-17180674
]
Hadoop QA commented on HDFS-15510:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
35s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF Lic
[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS Update (write-in-place)
[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180681#comment-17180681 ] Dinesh S. Atreya commented on HDFS-9607: Now Hadoop Distributed Data Store is an active effort, starting with _Hadoop Distributed Storage Layer (HDSL)_ HDFS-7240, *Ozone File System* (aka Hadoop Distributed Data Store) HDFS-13074 after formally being named with *Ozone: Rename HDSL to HDDS* HDFS-13405. Ozone is built on a highly available, replicated block storage layer called Hadoop Distributed Data Store (HDDS) and is S3 compatible. Ozone can function effectively in containerized environments such as _*Kubernetes*_ and _*YARN*_. Ozone supports different protocols like S3 and Hadoop File System APIs. Work on this JIRA and related JIRAs are using S3 compatible Ozone as a base. > Advance Hadoop Architecture (AHA) - HDFS Update (write-in-place) > > > Key: HDFS-9607 > URL: https://issues.apache.org/jira/browse/HDFS-9607 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Dinesh S. Atreya >Priority: Major > > Link to Umbrella JIRA > https://issues.apache.org/jira/browse/HADOOP-12620 > Provide capability to carry out in-place writes/updates. Only writes in-place > are supported where the existing length does not change. > For example, "Hello World" can be replaced by "Hello HDFS!" > See > https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300 > for more details. > Currently following are supported. > # Sequential writes > # Append (HADOOP-1700, HDFS-265) > # Snapshots (HDFS-2802) > # Truncate (HDFS-3107) > This JIRA is for random updates (write-in-place). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180695#comment-17180695
]
Íñigo Goiri commented on HDFS-15535:
I tend to prefer something like [^HDFS-15535-05.patch] with the class.
However, it looks to me like {{RemoteResult}} already does everything you are
doing here.
As this is parametrized you dont need to cast
{code}
RemoteResult response =
rpcClient.invokeSequential(method, locations, String.class, null);
RemoteLocation loc= response.getLoc();
result = response.getResult();
result = invokedResult.replaceFirst(loc.getDest(), loc.getSrc());
{code}
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS Update (write-in-place)
[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180681#comment-17180681 ] Dinesh S. Atreya edited comment on HDFS-9607 at 8/19/20, 5:13 PM: -- Now Hadoop Distributed Data Store is an active effort, starting with _Hadoop Distributed Storage Layer (HDSL)_ HDFS-7240, *Ozone File System* (aka Hadoop Distributed Data Store) HDFS-13074 after formally being named with *Ozone: Rename HDSL to HDDS* HDFS-13405. Ozone is built on a highly available, replicated block storage layer called Hadoop Distributed Data Store (HDDS) and is S3 compatible. Ozone can function effectively in containerized environments such as _*Kubernetes*_ and _*YARN*_. Ozone supports different protocols like S3 and Hadoop File System APIs. Work on this JIRA and related JIRAs are using HDDS/Ozone as a base given HDFS enhancements to work on top of HDDS. was (Author: dinatreya): Now Hadoop Distributed Data Store is an active effort, starting with _Hadoop Distributed Storage Layer (HDSL)_ HDFS-7240, *Ozone File System* (aka Hadoop Distributed Data Store) HDFS-13074 after formally being named with *Ozone: Rename HDSL to HDDS* HDFS-13405. Ozone is built on a highly available, replicated block storage layer called Hadoop Distributed Data Store (HDDS) and is S3 compatible. Ozone can function effectively in containerized environments such as _*Kubernetes*_ and _*YARN*_. Ozone supports different protocols like S3 and Hadoop File System APIs. Work on this JIRA and related JIRAs are using S3 compatible Ozone as a base. > Advance Hadoop Architecture (AHA) - HDFS Update (write-in-place) > > > Key: HDFS-9607 > URL: https://issues.apache.org/jira/browse/HDFS-9607 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Dinesh S. Atreya >Priority: Major > > Link to Umbrella JIRA > https://issues.apache.org/jira/browse/HADOOP-12620 > Provide capability to carry out in-place writes/updates. Only writes in-place > are supported where the existing length does not change. > For example, "Hello World" can be replaced by "Hello HDFS!" > See > https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300 > for more details. > Currently following are supported. > # Sequential writes > # Append (HADOOP-1700, HDFS-265) > # Snapshots (HDFS-2802) > # Truncate (HDFS-3107) > This JIRA is for random updates (write-in-place). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9814) Advance Hadoop Architecture (AHA) - HDFS Block Level Leases (Locks)
[ https://issues.apache.org/jira/browse/HDFS-9814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180707#comment-17180707 ] Dinesh S. Atreya commented on HDFS-9814: Now Hadoop Distributed Data Store is an active effort, starting with _Hadoop Distributed Storage Layer (HDSL)_ HDFS-7240, *Ozone File System* (aka Hadoop Distributed Data Store) HDFS-13074 after formally being named with *Ozone: Rename HDSL to HDDS* HDFS-13405. Ozone is built on a highly available, replicated block storage layer called Hadoop Distributed Data Store (HDDS) and is S3 compatible. Ozone can function effectively in containerized environments such as _*Kubernetes*_ and _*YARN*_. Ozone supports different protocols like S3 and Hadoop File System APIs. Work on this JIRA and related JIRAs are using HDDS/Ozone as a base given HDFS enhancements to work on top of HDDS. > Advance Hadoop Architecture (AHA) - HDFS Block Level Leases (Locks) > --- > > Key: HDFS-9814 > URL: https://issues.apache.org/jira/browse/HDFS-9814 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Dinesh S. Atreya >Priority: Major > > Link to Umbrella JIRA https://issues.apache.org/jira/browse/HADOOP-12620 > See > https://issues.apache.org/jira/browse/HADOOP-12620?focusedCommentId=15046300&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15046300 > for more details. > Currently, only file level leases (locks) are supported. This JIRA is for > expanding the capability to allow block level leases (locks). > Block level leases presume that Block Manager as a Service (reference JIRA > https://issues.apache.org/jira/browse/HDFS-5477) solution is available. > File level leases are default. When block level leases are requested, they > will supersede file level leases and lease (lock) capability will advance to > block levels. In the absence of block level leases, leases degenerate into > file level leases. > If we correspond to DBMS paradigm, file level leases amount to table locks; > while row level locks are not possible with HDFS, the closest we can get to > is block level leases (locks). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-06.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180741#comment-17180741
]
Íñigo Goiri commented on HDFS-15535:
Thanks [~ayushtkn] for [^HDFS-15535-06.patch], this integrates much better.
I have the feeling that we should do:
{code}
@SuppressWarnings("unchecked")
T ret = (T)result;
return new RemoteResult(loc, ret);
{code}
Then to avoid churn, can we leave loc as loc instead of using the new
invokedLocation var?
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180755#comment-17180755
]
Ayush Saxena commented on HDFS-15535:
-
Thanx [~elgoiri]
Can do this :
{code:java}
@SuppressWarnings("unchecked")
R ret = (R) result;
return new RemoteResult<>(loc, ret);{code}
or :
{code:java}
@SuppressWarnings("unchecked")
R ret = (R) result;
return new RemoteResult((T) loc, ret);
{code}
loc needs to be cast to T. Let me know which looks good
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180759#comment-17180759
]
Íñigo Goiri commented on HDFS-15535:
Yes, makes sense to cast both, either one is fine with me.
I guess he most consistent one is to do the cast outside for both.
{code}
@SuppressWarnings("unchecked")
T loc = (T)location;
@SuppressWarnings("unchecked")
R ret = (R) result;
return new RemoteResult<>(loc, ret);
{code}
I'm fine either way.
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180761#comment-17180761
]
Hadoop QA commented on HDFS-15535:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 32s{color}
| {color:red} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warni
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-07.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180770#comment-17180770
]
Ayush Saxena commented on HDFS-15535:
-
Have updated, with the Casting both params.
For the case when first result is sent, in that case {{locations.get(0)}}
doesn't require a cast, since {{locations}} is itself of type T. That didn't
bug me for cast even when I did :
{{return new RemoteResult(locations.get(0), ret);}}
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15533) Provide DFS API compatible class(ViewDistributedFileSystem), but use ViewFileSystemOverloadScheme inside
[
https://issues.apache.org/jira/browse/HDFS-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-15533:
---
Summary: Provide DFS API compatible class(ViewDistributedFileSystem), but
use ViewFileSystemOverloadScheme inside (was: Provide DFS API compatible
class, but use ViewFileSystemOverloadScheme inside)
> Provide DFS API compatible class(ViewDistributedFileSystem), but use
> ViewFileSystemOverloadScheme inside
>
>
> Key: HDFS-15533
> URL: https://issues.apache.org/jira/browse/HDFS-15533
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: dfs, viewfs
>Affects Versions: 3.4.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Fix For: 3.4.0
>
>
> I have been working on a thought from last week is that, we wanted to provide
> DFS compatible APIs with mount functionality. So, that existing DFS
> applications can work with out class cast issues.
> When we tested with other components like Hive and HBase, I noticed some
> classcast issues.
> {code:java}
> HBase example:
> java.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystemjava.lang.ClassCastException:
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme cannot be cast to
> org.apache.hadoop.hdfs.DistributedFileSystem at
> org.apache.hadoop.hbase.util.FSUtils.getDFSHedgedReadMetrics(FSUtils.java:1748)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerWrapperImpl.(MetricsRegionServerWrapperImpl.java:146)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1594)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1001)
> at java.lang.Thread.run(Thread.java:748){code}
> {code:java}
> Hive:
> |io.AcidUtils|: Failed to get files with ID; using regular API: Only
> supported for DFS; got class
> org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme{code}
> SO, here the implementation details are like follows:
> We extended DistributedFileSystem and created a class called "
> ViewDistributedFileSystem"
> This vfs=ViewFirstibutedFileSystem, try to initialize
> ViewFileSystemOverloadScheme. If success call will delegate to vfs. If fails
> to initialize due to no mount points, or other errors, it will just fallback
> to regular DFS init. If users does not configure any mount, system will
> behave exactly like today's DFS. If there are mount points, vfs functionality
> will come under DFS.
> I have a patch and will post it in some time.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180787#comment-17180787
]
Íñigo Goiri commented on HDFS-15535:
I just realized was this broken before? We were casting the ret to T before.
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180791#comment-17180791
]
Ayush Saxena commented on HDFS-15535:
-
No, Earlier there was only {{T}} which was the return type, Now we have two
{{T}} for location and {{R}} for result.
{code:java}
public T invokeSequential(
final List locations,
final RemoteMethod remoteMethod, Class expectedResultClass,
{code}
Earlier {{expectedResultClass}} had , so that was correct. If this was
broken in any case, Almost all non-multiple dest calls would have failed. :-)
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180807#comment-17180807
]
Hadoop QA commented on HDFS-15535:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 31s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
29s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
20s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF Lic
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180904#comment-17180904
]
Íñigo Goiri commented on HDFS-15535:
Oh, I see, my bad.
Just for simplicity and to keep it in sync with the other code, let's make T be
the result type and R the location then.
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena updated HDFS-15535:
Attachment: HDFS-15535-08.patch
> RBF: Fix Namespace path to snapshot path resolution for snapshot API
>
>
> Key: HDFS-15535
> URL: https://issues.apache.org/jira/browse/HDFS-15535
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-15535-01.patch, HDFS-15535-02.patch,
> HDFS-15535-03.patch, HDFS-15535-04.patch, HDFS-15535-05.patch,
> HDFS-15535-06.patch, HDFS-15535-07.patch, HDFS-15535-08.patch
>
>
> Presently, after invoking {{createSnapshot}} and {{getSnapshotListing}}, the
> namespace path is replaced with mount path.
> This presumes as of now that, the invokedLocation shall always be the first
> one in the sequence, but there are multiple reasons, where the directory
> might not be in the first location.
> So, rather than replacing using firstLocation, we should replace path using
> actual invoked Location.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
[
https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Fei Hui updated HDFS-14852:
---
Attachment: HDFS-14852.006.patch
> Remove of LowRedundancyBlocks do NOT remove the block from all queues
> -
>
> Key: HDFS-14852
> URL: https://issues.apache.org/jira/browse/HDFS-14852
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: CorruptBlocksMismatch.png, HDFS-14852.001.patch,
> HDFS-14852.002.patch, HDFS-14852.003.patch, HDFS-14852.004.patch,
> HDFS-14852.005.patch, HDFS-14852.006.patch, screenshot-1.png
>
>
> LowRedundancyBlocks.java
> {code:java}
> // Some comments here
> if(priLevel >= 0 && priLevel < LEVEL
> && priorityQueues.get(priLevel).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
> + " from priority queue {}",
> block, priLevel);
> decrementBlockStat(block, priLevel, oldExpectedReplicas);
> return true;
> } else {
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> for (int i = 0; i < LEVEL; i++) {
> if (i != priLevel && priorityQueues.get(i).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
> " {} from priority queue {}", block, i);
> decrementBlockStat(block, i, oldExpectedReplicas);
> return true;
> }
> }
> }
> return false;
> }
> {code}
> Source code is above, the comments as follow
> {quote}
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> {quote}
> The function "remove" does NOT remove the block from all queues.
> Function add from LowRedundancyBlocks.java is used on some places and maybe
> one block in two or more queues.
> We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is
> related to this.
> Upload initial patch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14852) Remove of LowRedundancyBlocks do NOT remove the block from all queues
[
https://issues.apache.org/jira/browse/HDFS-14852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180967#comment-17180967
]
Fei Hui commented on HDFS-14852:
[~sodonnell] Upload v006 patch with your suggestion， Please review
> Remove of LowRedundancyBlocks do NOT remove the block from all queues
> -
>
> Key: HDFS-14852
> URL: https://issues.apache.org/jira/browse/HDFS-14852
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.2.0, 3.0.3, 3.1.2, 3.3.0
>Reporter: Fei Hui
>Assignee: Fei Hui
>Priority: Major
> Attachments: CorruptBlocksMismatch.png, HDFS-14852.001.patch,
> HDFS-14852.002.patch, HDFS-14852.003.patch, HDFS-14852.004.patch,
> HDFS-14852.005.patch, HDFS-14852.006.patch, screenshot-1.png
>
>
> LowRedundancyBlocks.java
> {code:java}
> // Some comments here
> if(priLevel >= 0 && priLevel < LEVEL
> && priorityQueues.get(priLevel).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block {}"
> + " from priority queue {}",
> block, priLevel);
> decrementBlockStat(block, priLevel, oldExpectedReplicas);
> return true;
> } else {
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> for (int i = 0; i < LEVEL; i++) {
> if (i != priLevel && priorityQueues.get(i).remove(block)) {
> NameNode.blockStateChangeLog.debug(
> "BLOCK* NameSystem.LowRedundancyBlock.remove: Removing block" +
> " {} from priority queue {}", block, i);
> decrementBlockStat(block, i, oldExpectedReplicas);
> return true;
> }
> }
> }
> return false;
> }
> {code}
> Source code is above, the comments as follow
> {quote}
> // Try to remove the block from all queues if the block was
> // not found in the queue for the given priority level.
> {quote}
> The function "remove" does NOT remove the block from all queues.
> Function add from LowRedundancyBlocks.java is used on some places and maybe
> one block in two or more queues.
> We found that corrupt blocks mismatch corrupt files on NN web UI. Maybe it is
> related to this.
> Upload initial patch
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15535) RBF: Fix Namespace path to snapshot path resolution for snapshot API
[
https://issues.apache.org/jira/browse/HDFS-15535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180969#comment-17180969
]
Hadoop QA commented on HDFS-15535:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 30m
47s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
9s{color} | {color:blue} Used deprecated FindBugs config; considering switching
to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
1s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF Lic
