[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471461#comment-16471461
]
genericqa commented on HDFS-13346:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m
2s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922946/HDFS-13346.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 029b993546fb 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d76fbbc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24177/testReport/ |
| Max. process+thread count | 960 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24177/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL:
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471405#comment-16471405
]
Yiqun Lin commented on HDFS-13346:
--
Thanks [~elgoiri] for the review.
{quote}
In HDFS-13346.005.patch, I would prefer using a Whitebox instead of having to
expose the quota module but I'm OK with it.
{quote}
Had used Whitebox in the latest patch.
Attach the patch and pending Jenkins.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Labels: incompatible
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch, HDFS-13346.004.patch, HDFS-13346.005.patch,
> HDFS-13346.006.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16470674#comment-16470674
]
Íñigo Goiri commented on HDFS-13346:
In [^HDFS-13346.005.patch], I would prefer using a Whitebox instead of having
to expose the quota module but I'm OK with it.
The unit tests pass and the report looks good; +1.
As [~linyiqun] is more familiar with the quota path, I'll leave the final
review and commit to him.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Labels: incompatible
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch, HDFS-13346.004.patch, HDFS-13346.005.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16469854#comment-16469854
]
Yiqun Lin commented on HDFS-13346:
--
[~elgoiri], would you mind having a look for this?
[~liuhongtong], does this patch address your case now?
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Labels: incompatible
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch, HDFS-13346.004.patch, HDFS-13346.005.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468520#comment-16468520
]
genericqa commented on HDFS-13346:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
38s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 82m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922599/HDFS-13346.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2adf02d94408 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8091350 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24160/testReport/ |
| Max. process+thread count | 963 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24160/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
>
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16468466#comment-16468466
]
Yiqun Lin commented on HDFS-13346:
--
Attach the patch to fix failed unit test.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Labels: incompatible
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch, HDFS-13346.004.patch, HDFS-13346.005.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467335#comment-16467335
]
genericqa commented on HDFS-13346:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 39s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 29s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd |
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12922438/HDFS-13346.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2831ba0d9717 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7450583 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/24149/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/24149/testReport/ |
| Max. process+thread count | 1364 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/24149/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465667#comment-16465667
]
Yiqun Lin commented on HDFS-13346:
--
Thanks [~liuhongtong]. Will attach the patch soon, :).
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: Yiqun Lin
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465579#comment-16465579
]
liuhongtong commented on HDFS-13346:
[~linyiqun] I am encountering compiling issues in latest source code and
failure in unit test.
I will be pleased to transfer this issue to you. Thanks.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465552#comment-16465552
]
Yiqun Lin commented on HDFS-13346:
--
Hi [~liuhongtong], are you still working on this? If not, I can continue the
work on this. It would be a good fix for the quota supported in RBF.
Thanks
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435018#comment-16435018
]
Yiqun Lin commented on HDFS-13346:
--
Hi [~liuhongtong], I did a quick look for your patch. I'm afraid you are
missing my previous proposal, :P.
{quote}So I mean following way should be still okay:
{noformat}
One better way is that we invoke update operation this.rpcServer.setQuota in
RouterAdminServer, not in RouterQuotaUpdateService.
{noformat}
liuhongtong, I think you can go ahead by this way, .
{quote}
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch,
> HDFS-13346.003.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434949#comment-16434949
]
genericqa commented on HDFS-13346:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 30m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 5s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12918668/HDFS-13346.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 421aeb2cc75f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c7cd362 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23893/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23893/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429960#comment-16429960
]
liuhongtong commented on HDFS-13346:
Hi [~linyiqun], I am working on this issue.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425262#comment-16425262
]
Yiqun Lin commented on HDFS-13346:
--
Hi [~liuhongtong], any updating for this? If you aren't in free, I will give a
try for improving this, :). Please let me know.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416762#comment-16416762
]
Yiqun Lin commented on HDFS-13346:
--
Thanks [~elgoiri] for sharing the comments.
{quote}
You would have the same issue if A sets it to 200 and B later sets it to 300.
You have two admins doing conflicting operations.
{quote}
Agreed. Actually these should be the conflicting operations. Even though one
admin set quota successfully, but after this the other admin do the new
updating. The previous admin will see the false quota value. So I mean
following way should be still okay:
{code}
One better way is that we invoke update operation this.rpcServer.setQuota in
RouterAdminServer, not in RouterQuotaUpdateService.
{code}
[~liuhongtong], I think you can go ahead by this way, :).
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415936#comment-16415936
]
Íñigo Goiri commented on HDFS-13346:
I'm not sure there is an atomicity problem in the example described.
You would have the same issue if A sets it to 200 and B later sets it to 300.
You have two admins doing conflicting operations.
We can get fancy and lock the entries but if after the lock, B comes sets it to
300, you have the same issue.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415277#comment-16415277
]
Yiqun Lin commented on HDFS-13346:
--
[~liuhongtong], I think this will be a problem. I am thinking another way: Can
we define a new field {{isQuotaUpdated}} indicates that the router quota was
updated, and we set quota for subclusters in {{RouterQuotaUpdateService}} if
check this flag be updated.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415229#comment-16415229
]
liuhongtong commented on HDFS-13346:
[~linyiqun] Thanks for your recommendation.
RouterAdminServer invokes this.rpcServer.setQuota to set Quota of Name Service
and then updates or adds MountTableEntry to State Store.
But I am worried about how to ensure consistency and atomicity between
different Routers.
RouterAdmin A sets nsQuota 200 | RouterAdmin B sets
nsQuota 300
| |
|
RouterAdminServer A set NS nsQuota 200 | |
| |
RouterAdminServer B set NS nsQuota 300
| |
RouterAdminServer B update State Store
RouterAdminServer A update State Store | |
Now Router nsQuota is 200, but Name Service nsQuota is 300.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413300#comment-16413300
]
Yiqun Lin commented on HDFS-13346:
--
Hi [~liuhongtong], thanks for filling this! Just take a quick look, current
change won't fix this. {{currentQuotaUsage}} won't get ns quota value of
subclusters. It's constructed by following codes:
{code:java}
QuotaUsage.Builder builder = new QuotaUsage.Builder()
.fileAndDirectoryCount(nsCount).spaceConsumed(ssCount);
if (hasQuotaUnSet) {
builder.quota(HdfsConstants.QUOTA_DONT_SET);
}
{code}
It's only used to check if there are some subcluster which don't have quota
set. Then we do the initial setting.
One better way is that we invoke update operation {{this.rpcServer.setQuota}}
in {{RouterAdminServer}}, not in {{RouterQuotaUpdateService}}.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: liuhongtong
>Assignee: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch, HDFS-13346.002.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413075#comment-16413075
]
genericqa commented on HDFS-13346:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 18s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.federation.router.TestRouterQuota |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916097/HDFS-13346.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 48db3ed187ae 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 28790b8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/23664/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23664/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| Test Results |
[jira] [Commented] (HDFS-13346) RBF: Fix synchronization of router quota and ns quota
[
https://issues.apache.org/jira/browse/HDFS-13346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16413049#comment-16413049
]
genericqa commented on HDFS-13346:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} HDFS-13346 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-13346 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916096/HDFS-13346.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23663/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> RBF: Fix synchronization of router quota and ns quota
> -
>
> Key: HDFS-13346
> URL: https://issues.apache.org/jira/browse/HDFS-13346
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: liuhongtong
>Priority: Major
> Attachments: HDFS-13346.001.patch
>
>
> Check Router Quota and ns Quota:
> {code}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 150/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Update Router Quota:
> {code:java}
> $ hdfs dfsrouteradmin -setQuota /ns10t -nsQuota 400
> Successfully set quota for mount point /ns10t
> {code}
> Check Router Quota and ns Quota:
> {code:java}
> $ hdfs dfsrouteradmin -ls /ns10t
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /ns10tns10->/ns10t hadp
> hadp rwxr-xr-x [NsQuota: 400/319,
> SsQuota: -/-]
> /ns10t/ns1mountpoint ns1->/a/tthadp
> hadp rwxr-xr-x [NsQuota: -/-, SsQuota:
> -/-]
> $ hdfs dfs -count -q hdfs://ns10/ns10t
> 150-155none inf3
> 302 0 hdfs://ns10/ns10t
> {code}
> Now Router Quota has updated successfully, but ns Quota not.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
