jerryshao opened a new issue, #13420:
URL: https://github.com/apache/gravitino/issues/13420
### Version
main branch
### Describe what's wrong
`TestJobManagerMultiNode#testCancelJobFromAnotherNode` expects the stored
job status to still be `CANCELLING` after the first status pull on the owner
node:
```java
nodeA.pullAndUpdateJobStatus();
Awaitility.await()...until(() -> executorA.getJobStatus(...) ==
JobHandle.Status.CANCELLED);
Assertions.assertEquals(JobHandle.Status.CANCELLING,
getJob(job.name()).status());
```
A single pull can already store `CANCELLED`. In
`JobManager#pullAndUpdateOwnedJobStatus`, a `CANCELLING` job owned by a
node-local executor goes through `cancelOwnedJob`, which cancels the job and
then immediately reads its status back:
```java
jobExecutor.cancelJob(job.jobExecutionId());
return jobExecutor.getJobStatus(job.jobExecutionId());
```
If the `LocalJobExecutor` process exits before that `getJobStatus` call, the
pull writes `CANCELLED`, and the assertion fails. Whether it fails depends only
on how fast the process dies. The behaviour of `JobManager` itself is correct;
only the test assumes cancellation is slower than one pull.
### Error message and/or stacktrace
```
TestJobManagerMultiNode > testCancelJobFromAnotherNode() > [POSTGRESQL
Backend] FAILED
org.opentest4j.AssertionFailedError: expected: <CANCELLING> but was:
<CANCELLED>
at
org.apache.gravitino.job.TestJobManagerMultiNode.testCancelJobFromAnotherNode(TestJobManagerMultiNode.java:155)
```
### How to reproduce
Intermittent in CI; seen on the PostgreSQL backend while the H2 and MySQL
variants passed in the same run. It reproduces whenever the cancelled shell
process exits before `cancelOwnedJob` reads the status back.
### Additional context
Suggested fix: after the first pull, accept either `CANCELLING` or
`CANCELLED`, and keep the final assertions (`CANCELLED` plus `finishedAt > 0`)
after the second pull.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
