[
https://issues.apache.org/jira/browse/GEODE-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119892#comment-17119892
]
Jinmei Liao commented on GEODE-8200:
------------------------------------
If a locator that initiated the rebalance operation went down before it can
record the "completed" state of the operation, then that operation status will
be "orphaned". i.e. the executor died, there is no one to execute the
"whenComplete" section of the code here:
https://github.com/apache/geode/blob/57cc3c7b40816bc1b7bbae80481dea608c7caff5/geode-core/src/main/java/org/apache/geode/management/internal/operation/OperationManager.java#L70
That operation id will be stuck in "in_progress" status and kept in the history
forever, because apparently we only clean the "done" record.
> Rebalance operations stuck in "IN_PROGRESS" state forever
> ---------------------------------------------------------
>
> Key: GEODE-8200
> URL: https://issues.apache.org/jira/browse/GEODE-8200
> Project: Geode
> Issue Type: Bug
> Components: management
> Reporter: Aaron Lindsey
> Priority: Major
> Labels: GeodeOperationAPI
>
> We use the management REST API to call rebalance immediately before stopping
> a server to limit the possibility of data loss. In a cluster with 3 locators,
> 3 servers, and no regions, we noticed that sometimes the rebalance operation
> never ends if one of the locators is restarting concurrently with the
> rebalance operation.
> More specifically, the scenario where we see this issue crop up is during an
> automated "rolling restart" operation in a Kubernetes environment which
> proceeds as follows:
> * At most one locator and one server are restarting at any point in time
> * Each locator/server waits until the previous locator/server is fully online
> before restarting
> * Immediately before stopping a server, a rebalance operation is performed
> and the server is not stopped until the rebalance operation is completed
> The impact of this issue is that the "rolling restart" operation will never
> complete, because it cannot proceed with stopping a server until the
> rebalance operation is completed. A human is then required to intervene and
> manually trigger a rebalance and stop the server. This type of "rolling
> restart" operation is triggered fairly often in Kubernetes — any time part of
> the configuration of the locators or servers changes.
> The following JSON is a sample response from the management REST API that
> shows the rebalance operation stuck in "IN_PROGRESS".
> {code}
> {
> "statusCode": "IN_PROGRESS",
> "links": {
> "self":
> "http://geodecluster-sample-locator.default/management/v1/operations/rebalances/a47f23c8-02b3-443c-a367-636fd6921ea7";,
> "list":
> "http://geodecluster-sample-locator.default/management/v1/operations/rebalances";
> },
> "operationStart": "2020-05-27T22:38:30.619Z",
> "operationId": "a47f23c8-02b3-443c-a367-636fd6921ea7",
> "operation": {
> "simulate": false
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
