zhangshenghang opened a new issue, #13316:
URL: https://github.com/apache/seatunnel/issues/13316
### What happened
If `JobMaster.cleanJob()` throws, `jobMasterCompleteFuture` is never
completed and `JobMaster.run()`'s finally blocks on it forever: the coordinator
thread leaks, clients hang, and job state map entries are never cleaned.
Chain (dev @ 89f91a89af):
1. `JobMaster.initStateFuture` (`JobMaster.java:489-502`) registers
`jobStatusFuture.whenComplete(withTryCatch(LOGGER, (v, t) -> { ...; cleanJob();
jobMasterCompleteFuture.complete(jobResult); }))`.
2. `cleanJob()` (`JobMaster.java:1041-1046`) performs checkpoint-storage
deletes (`clearCheckpointIfNeed`) and job-history IMap writes
(`storeJobInfo`/`storeFinishedJobState`) — all of which can throw (IO errors,
`HazelcastInstanceNotActiveException` during shutdown, storage backend outage).
3. `withTryCatch` is Hazelcast's log-and-swallow utility, so the exception
never escapes the callback — but `jobMasterCompleteFuture.complete(...)` is
skipped because it runs *after* `cleanJob()`.
4. `JobMaster.run()`'s finally does `jobMasterCompleteFuture.join()`
(`JobMaster.java:766-777`) — **permanent block**: the coordinator-pool thread
running `run()` is never released (`runningJobMasterMap` cleanup never happens
→ thread/memory leak), clients hang in `waitForJobComplete`, and
`scheduleRemoveJobStateMaps` never runs (job state map entries leak until the
next master switchover).
### Trigger
Any exception from `cleanJob()` at job end — e.g. checkpoint storage briefly
unavailable while deleting checkpoint data, or IMap unreachable during cluster
shutdown.
### Impact
Coordinator thread and JobMaster entry leak per occurrence (can exhaust the
coordinator pool), client CLI/REST hangs forever, job state maps leak.
### Suggested fix
Wrap `cleanJob()` in its own try/catch (log the failure) and complete
`jobMasterCompleteFuture` unconditionally — e.g. move the completion into a
finally, so cleanup failures degrade to warnings instead of hanging the job's
completion.
### SeaTunnel Version
dev (89f91a89af), Zeta engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]