[
https://issues.apache.org/jira/browse/GEODE-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145087#comment-17145087
]
ASF GitHub Bot commented on GEODE-8200:
---------------------------------------
kirklund commented on a change in pull request #5299:
URL: https://github.com/apache/geode/pull/5299#discussion_r445693168
##########
File path:
geode-junit/src/main/java/org/apache/geode/test/junit/rules/gfsh/GfshRule.java
##########
@@ -199,4 +220,23 @@ private void stopMembers(GfshExecution gfshExecution) {
}
execute(GfshScript.of(stopMemberScripts).withName("Stop-Members"));
}
+
+ public static String startServerCommand(String name, int port, int
connectedLocatorPort) {
Review comment:
There are actually a couple existing classes like this that we could
move to geode-junit and make any necessary changes:
```
geode-core/src/integrationTest/java/org/apache/geode/distributed/LocatorCommand.java
geode-core/src/integrationTest/java/org/apache/geode/distributed/ServerCommand.java
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Rebalance operations stuck in "IN_PROGRESS" state forever
> ---------------------------------------------------------
>
> Key: GEODE-8200
> URL: https://issues.apache.org/jira/browse/GEODE-8200
> Project: Geode
> Issue Type: Bug
> Components: management
> Reporter: Aaron Lindsey
> Assignee: Jianxia Chen
> Priority: Major
> Labels: GeodeOperationAPI
> Attachments: GEODE-8200-exportedLogs.zip
>
>
> We use the management REST API to call rebalance immediately before stopping
> a server to limit the possibility of data loss. In a cluster with 3 locators,
> 3 servers, and no regions, we noticed that sometimes the rebalance operation
> never ends if one of the locators is restarting concurrently with the
> rebalance operation.
> More specifically, the scenario where we see this issue crop up is during an
> automated "rolling restart" operation in a Kubernetes environment which
> proceeds as follows:
> * At most one locator and one server are restarting at any point in time
> * Each locator/server waits until the previous locator/server is fully online
> before restarting
> * Immediately before stopping a server, a rebalance operation is performed
> and the server is not stopped until the rebalance operation is completed
> The impact of this issue is that the "rolling restart" operation will never
> complete, because it cannot proceed with stopping a server until the
> rebalance operation is completed. A human is then required to intervene and
> manually trigger a rebalance and stop the server. This type of "rolling
> restart" operation is triggered fairly often in Kubernetes — any time part of
> the configuration of the locators or servers changes.
> The following JSON is a sample response from the management REST API that
> shows the rebalance operation stuck in "IN_PROGRESS".
> {code}
> {
> "statusCode": "IN_PROGRESS",
> "links": {
> "self":
> "http://geodecluster-sample-locator.default/management/v1/operations/rebalances/a47f23c8-02b3-443c-a367-636fd6921ea7";,
> "list":
> "http://geodecluster-sample-locator.default/management/v1/operations/rebalances";
> },
> "operationStart": "2020-05-27T22:38:30.619Z",
> "operationId": "a47f23c8-02b3-443c-a367-636fd6921ea7",
> "operation": {
> "simulate": false
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
