zhangshenghang opened a new issue, #13316:
URL: https://github.com/apache/seatunnel/issues/13316

   ### What happened
   
   If `JobMaster.cleanJob()` throws, `jobMasterCompleteFuture` is never 
completed and `JobMaster.run()`'s finally blocks on it forever: the coordinator 
thread leaks, clients hang, and job state map entries are never cleaned.
   
   Chain (dev @ 89f91a89af):
   
   1. `JobMaster.initStateFuture` (`JobMaster.java:489-502`) registers 
`jobStatusFuture.whenComplete(withTryCatch(LOGGER, (v, t) -> { ...; cleanJob(); 
jobMasterCompleteFuture.complete(jobResult); }))`.
   2. `cleanJob()` (`JobMaster.java:1041-1046`) performs checkpoint-storage 
deletes (`clearCheckpointIfNeed`) and job-history IMap writes 
(`storeJobInfo`/`storeFinishedJobState`) — all of which can throw (IO errors, 
`HazelcastInstanceNotActiveException` during shutdown, storage backend outage).
   3. `withTryCatch` is Hazelcast's log-and-swallow utility, so the exception 
never escapes the callback — but `jobMasterCompleteFuture.complete(...)` is 
skipped because it runs *after* `cleanJob()`.
   4. `JobMaster.run()`'s finally does `jobMasterCompleteFuture.join()` 
(`JobMaster.java:766-777`) — **permanent block**: the coordinator-pool thread 
running `run()` is never released (`runningJobMasterMap` cleanup never happens 
→ thread/memory leak), clients hang in `waitForJobComplete`, and 
`scheduleRemoveJobStateMaps` never runs (job state map entries leak until the 
next master switchover).
   
   ### Trigger
   
   Any exception from `cleanJob()` at job end — e.g. checkpoint storage briefly 
unavailable while deleting checkpoint data, or IMap unreachable during cluster 
shutdown.
   
   ### Impact
   
   Coordinator thread and JobMaster entry leak per occurrence (can exhaust the 
coordinator pool), client CLI/REST hangs forever, job state maps leak.
   
   ### Suggested fix
   
   Wrap `cleanJob()` in its own try/catch (log the failure) and complete 
`jobMasterCompleteFuture` unconditionally — e.g. move the completion into a 
finally, so cleanup failures degrade to warnings instead of hanging the job's 
completion.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to