davidzollo commented on PR #12027: URL: https://github.com/apache/seatunnel/pull/12027#issuecomment-5549660009
### CI update for head `bcf9ad1` — correcting my previous analysis: the restore-retry fix is **not** the fix for this failure Run [33942761744](https://github.com/davidzollo/seatunnel/actions/runs/33942761744) is in, and it disproves the hypothesis I posted on `db88893`. - `engine-v2-it (11)`: **same failure, unchanged** — `testPendingJobNotDuplicatedAcrossRepeatedMasterFailover`, `expected: <FINISHED> but was: <PENDING> within 3 minutes` (`SplitClusterPendingJobLifecycleFailoverIT.java:399`). - `engine-v2-it (8)`: no signal — it died before reaching any test, on a Maven Central transport error (`com.google.auto:auto-common:jar:1.2: Connection reset`, build stopped at `-rf :seatunnel-config-shade`). Pure infrastructure, unrelated to this PR. So `73aaf491` ("Retry master-switch job restore on transient failure") does not change this outcome. Retracting the earlier claim that an abandoned restore future was the cause. #### What the run actually shows The restore path works. On the final master, the contested job's `JobMaster` is alive and re-applying for resources every ~3s for the whole 180s window: ``` 04:09:33 ResourceRequestHandler - Apply resource not success for job: 1148469947324760066, required: 1 slots, applied: 0 04:09:33 ResourceRequestHandler - request slot with retry error: ...NoEnoughResourceException 04:09:33 JobMaster - Pre resource application failed ... failed: 3/3 ... identical, every ~3s, through 04:13:01 (test gives up) ``` The job is not lost, not duplicated, and not stuck in restore. **It is starved of slots.** The timeline is what makes this damning: ``` 04:09:33.042 PhysicalPlan - Job pending_job_duplicate_dispatch_holder (…694530) state process is stopped 04:09:33.213 ResourceRequestHandler - Apply resource not success for job: …760066, required: 1 slots, applied: 0 ``` The holder job — which occupies all 4 of the worker's slots (`configurePendingLifecycleTest` sets `dynamicSlot=false`, `slotNum=4`) — is cancelled, and its slots are still not available 3.5 minutes later. The worker itself is healthy throughout: the only `AbstractResourceManager - Node heartbeat timeout` entries in the window name `[localhost]:5801` and `[localhost]:5802`, the two master nodes being cycled, never the worker `[localhost]:5803`. So the engine behaviour under test is: **slots held by a job that is cancelled after repeated master failovers are never returned, and any job waiting on them starves indefinitely.** That is a real defect, and it is arguably a more serious one than the duplicate-dispatch case this test was written for. #### Where I think it lives, and what I have not yet proven The release path narrows to `JobMaster#releasePipelineResource` (`seatunnel-engine-server/.../master/JobMaster.java:923`) and `JobMaster#releaseTaskGroupResource` (`:879`), which forward to `DefaultSlotService#releaseSlot` (`.../service/slot/DefaultSlotService.java:203`). That method rejects a release outright, with `WrongTargetSlotException`, in three cases (`:207`, `:212`, `:217`): the slot id is not in `assignedSlots`, the profile's `sequence` does not match the one the worker currently holds, or the profile's owner job id does not match. A restored `JobMaster` rebuilds its view from `ownedSlotProfilesIMap`, so a `SlotProfile` that has gone stale across four master epochs failing the `sequence` check at `:212` would produce exactly the observed end state — worker keeps the slot in `assignedSlots`, every subsequent `WorkerProfile` reports it as assigned, and no later coordinator can ever reclaim it. I want to be explicit that this last paragraph is a **hypothesis, not a proven root cause**. The E2E capture only retains WARN and above, and the decisive lines (`received slot release request …` at `DefaultSlotService.java:204`, and `release the pipeline %s resource` in `JobMaster#releasePipelineResource`) are INFO, so this run cannot tell me whether the release was attempted and rejected, or never attempted at all. I am not going to guess at a production change on that basis. #### Consequences for this PR 1. `73aaf491` should come out of this PR. It is an engine change in a `[Test][E2E]` PR that demonstrably does not fix the failure it was added for. The behaviour it addresses (`restoreAllRunningJobFromMasterNodeSwitch` submitted fire-and-forget with no exception handler, on a future no status-read path joins) still looks like a genuine robustness gap to me, but it belongs in its own `[Fix][Zeta]` PR with its own justification, not bundled here. 2. To get the decisive evidence I need one instrumented CI run that raises `DefaultSlotService` and `JobMaster` to INFO for this test, so the slot-release attempt is visible. I would rather spend one cycle on that than another on a guess. 3. I am **not** relaxing the assertion or extending the 180s timeout. The job is not slow; it is receiving `applied: 0` on every single attempt for the entire window, and a longer wait would not change that — it would only hide a real slot leak. Marking this blocking until the release path is pinned down. Reviewers: if the slot-release-across-epochs behaviour is already known and tracked elsewhere, please point me at it and I will rebase this test onto that fix rather than duplicating the investigation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
