[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16872034#comment-16872034
]
Hadoop QA commented on HDFS-14079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} HDFS-14079 does not apply to HDFS-13891. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14079 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949647/HDFS-14079-HDFS-13891.02.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/27070/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16724666#comment-16724666
]
Yiqun Lin commented on HDFS-14079:
--
As HDFS-13443 was merged, we can move forward with this, :).
[~surendrasingh], as I mentioned the consistency problem when doing the
fail-over in previous comment, I suppose we can add a test case now.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702880#comment-16702880
]
Surendra Singh Lilhore commented on HDFS-14079:
---
Thanks [~brahmareddy], [~linyiqun], [~elgoiri] for comments.
{quote}I am thinking one further problem. This will maybe lead a inconsistent
Router state.
{quote}
This problem will be solved in HDFS-13443.
{quote}This is interesting as it would go in the direction of HADOOP-15774
where we use the repos to discover instead of configuration files.
This might be a good point to start.
{quote}
Okay, I will try this...
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16702328#comment-16702328
]
Íñigo Goiri commented on HDFS-14079:
{quote}
Existing logic is depend on nameservice Id. If we want to reuse it, we have to
define nsid for router in one property and RouterAdmin will use it to get the
admin address list.
{quote}
We should just refactor the DFSUtil part to extract this, I think it's there
and it's just a matter of changing visibility.
{quote}
@ProtocolInfo(protocolName = HdfsConstants.CLIENT_NAMENODE_PROTOCOL_NAME, Not
sure whether this intened.Íñigo Goiri can you confirm same.Even this can be
done in seperate jira.
{quote}
This was originally a copy of what the NN did.
We may want to revisit this setting.
BTW, we need to take care of the Kerberos setting here (we should do this as
part of the security umbrella).
{quote}
IMO, Instead of exposing config for admin address(which needs additional
validation),RouterStateManager has admin address like below.SO we can use that?
{quote}
This is interesting as it would go in the direction of HADOOP-15774 where we
use the repos to discover instead of configuration files.
This might be a good point to start.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701856#comment-16701856
]
Brahma Reddy Battula commented on HDFS-14079:
-
[~surendrasingh] thanks for reporting and working on this.
* As [~linyiqun] pointed consistency will be problem.we might need to retry
till state store refreshed ( which is by default 60 Sec).looks you used
TRY_ONCE_THEN_FAIL. Please add testcases for that. Ideally admin operations
will be rare(add,ls,safemode..),May be because of this it was not implemented.?
* @ProtocolInfo(protocolName = HdfsConstants.CLIENT_NAMENODE_PROTOCOL_NAME,
Not sure whether this intened.[~elgoiri] can you confirm same.Even this can be
done in seperate jira.
* IMO, Instead of exposing config for admin address(which needs additional
validation),RouterStateManager has admin address like below.SO we can use that?
{code:java}
List cachedRecords =
router.getRouterStateManager().getCachedRecords();
String adminAddress = routerState.getAdminAddress();
{code}
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701471#comment-16701471
]
Surendra Singh Lilhore commented on HDFS-14079:
---
Thanks [~elgoiri] for review..
{quote}Instead of {{admin-address.list}}, for consistency with the NN say, we
may want to do the suffixes {{admin-address.r1}} and reuse all that logic to
get addresses.
{quote}
Existing logic is depend on nameservice Id. If we want to reuse it, we have to
define nsid for router in one property and RouterAdmin will use it to get the
admin address list.
Do you want me to write complete logic again without nsid and just use router
ID to get address ?
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700795#comment-16700795
]
Íñigo Goiri commented on HDFS-14079:
Thanks [^HDFS-14079-HDFS-13891.02.patch] for the patch.
* {{RouterAdminProtocal}} should be {{RouterAdminProtocol}}?
* Should we have ROUTER_ADMIN_PROTOCOL_NAME defined in HdfsConstants or leave
it as part of hadoop-hdfs-rbf?
* I would prefer to define the string as
{{RouterAdminProtocol.class.getName()}} even though the others do it
differently.
* Add a javadoc to {{RouterAdminProtocal}}.
* Instead of {{admin-address.list}}, for consistency with the NN say, we may
want to do the suffixes {{admin-address.r1}} and reuse all that logic to get
addresses.
[~virajith], you recently did the HA client for the alias map.
Can you take a pass?
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700313#comment-16700313
]
Hadoop QA commented on HDFS-14079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
45s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
55s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-hdfs-project: The patch generated 23 new
+ 7 unchanged - 0 fixed = 30 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-rbf generated 3 new +
0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
43s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 10s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 38s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | Useless control flow to next line in
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.enterSafeMode(List) At
RouterAdmin.java:line in
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.enterSafeMode(List) At
RouterAdmin.java:[line 848] |
| | Useless control flow to next line in
org.apache.hadoop.hdfs.tools.federation.RouterAdmin.getSafeMode(List) At
RouterAdmin.
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700112#comment-16700112
]
Surendra Singh Lilhore commented on HDFS-14079:
---
Attached v2, added missing files.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch,
> HDFS-14079-HDFS-13891.02.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700040#comment-16700040
]
Hadoop QA commented on HDFS-14079:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
37s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 2m
50s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 2m 50s{color}
| {color:red} hadoop-hdfs-project in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-hdfs-project: The patch generated 17 new
+ 7 unchanged - 0 fixed = 24 total (was 7) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
28s{color} | {color:red} hadoop-hdfs-project_hadoop-hdfs-rbf generated 14 new +
0 unchanged - 0 fixed = 14 total (was 0) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
31s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14079 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949611/HDFS-14079-HDFS-13891.01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname |
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16700039#comment-16700039
]
Yiqun Lin commented on HDFS-14079:
--
Hi [~surendrasingh], I am thinking one further problem. This will maybe lead a
inconsistent Router state.
For example, we have three Routers, R1, R2 and R3 as the admin list.
Firstly, we use R1 as the target Router, then we add a mount table m1 for R1.
Then R1 becomes unavailable, then client fail over to R2. We won't get m1 info
from current Router. I mean current implementation isn't fully HA endpoint.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699987#comment-16699987
]
Surendra Singh Lilhore commented on HDFS-14079:
---
Attached initial patch. Pls review.
# Added new property *"dfs.federation.router.admin-address.list"*, user can
configure comma separated router admin address. Client will fail over on
connection exception.
# If above property is not configured then it will
*"dfs.federation.router.admin-address"* property and connect with only one
router.
# With fail over option safemode command send rpc to random router, to solve
this problem added one option in safemode command *"-routers "*.
User can pass the list of router which he want to move in safe mode.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
> Attachments: HDFS-14079-HDFS-13891.01.patch
>
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692070#comment-16692070
]
Íñigo Goiri commented on HDFS-14079:
[~surendrasingh] for the solution that [~crh] is talking about, there is no
code change.
It would be a matter of putting the admin port behind a load balancer and
setting the config to point to that endpoint.
Anyway, we probably want to set a full HA endpoint in addition.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16691273#comment-16691273
]
Surendra Singh Lilhore commented on HDFS-14079:
---
Thansk [~crh] for working on this..
Pls can you upload your patch here..
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689939#comment-16689939
]
CR Hota commented on HDFS-14079:
[~surendrasingh] Thanks for reporting this. Internally I am trying to work on a
failover solution that is DNS based. Overall the approach is to hide 'N'
routers behind a single dns and handle randomization in the proxyfailover
layer. To begin with, this works fine without health checks till the ideal
solution based on health check is built and stabilized. Helps hide all router
endpoints and also add/remove routers easily.
Example dns we use that hides 8 different router instances.
{code:java}
crh-C02TC2FQG8WL:hadoop crh$ nslookup hdfs-router.dca1.uber.internal.
Server: 127.0.0.1
Address:127.0.0.1#53
Non-authoritative answer:
Name: hdfs-router.dca1.uber.internal
Address: 10.14.130.151
Name: hdfs-router.dca1.uber.internal
Address: 10.20.139.168
Name: hdfs-router.dca1.uber.internal
Address: 10.18.142.29
Name: hdfs-router.dca1.uber.internal
Address: 10.16.7.170
Name: hdfs-router.dca1.uber.internal
Address: 10.20.134.4
Name: hdfs-router.dca1.uber.internal
Address: 10.19.131.147
Name: hdfs-router.dca1.uber.internal
Address: 10.14.131.20
Name: hdfs-router.dca1.uber.internal
Address: 10.20.11.40
{code}
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14079) RBF: RouterAdmin should have failover concept for router
[
https://issues.apache.org/jira/browse/HDFS-14079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16689788#comment-16689788
]
Íñigo Goiri commented on HDFS-14079:
Thanks [~surendrasingh] for opening this; we should make this part of
HDFS-13891.
Regarding the JIRA itself, I was trying to compare to the NN but it doesn't
look like the NN has an admin protocol (I thought it did).
In any case, we can just make this be like the regular RPC client and use one
of the default failover approaches.
> RBF: RouterAdmin should have failover concept for router
>
>
> Key: HDFS-14079
> URL: https://issues.apache.org/jira/browse/HDFS-14079
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Surendra Singh Lilhore
>Assignee: Surendra Singh Lilhore
>Priority: Major
>
> Currenlty {{RouterAdmin}} connect with only one router for admin operation,
> if the configured router is down then router admin command is failing. It
> should allow to configure all the router admin address.
> {code}
> // Initialize RouterClient
> try {
> String address = getConf().getTrimmed(
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_KEY,
> RBFConfigKeys.DFS_ROUTER_ADMIN_ADDRESS_DEFAULT);
> InetSocketAddress routerSocket = NetUtils.createSocketAddr(address);
> client = new RouterClient(routerSocket, getConf());
> } catch (RPC.VersionMismatch v) {
> System.err.println(
> "Version mismatch between client and server... command aborted");
> return exitCode;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
