DanielLeens commented on PR #12943:
URL: https://github.com/apache/seatunnel/pull/12943#issuecomment-6793449092

   CI follow-up on head `16714019`: the failed job was re-run, and 
[`engine-resource-managers-it 
(17)`](https://github.com/goutamadwant/seatunnel/actions/runs/42765842651/job/130364594639)
 failed again on attempt 2, so my earlier "looks intermittent, just re-run" 
note no longer fully holds.
   
   - This time the failure is 
`KubernetesApplicationHaIT.applicationIdCancelRemovesRunningRecoveredHaApplication:484`
 (`ConditionTimeoutException ... within 3 minutes`). The first attempt failed 
in `masterPodLossKeepsNativeJobAndFixedWorkers{boolean}[2]`.
   - Both failures are the same scenario: the test deletes the **active** 
master Pod and then waits up to 180 seconds (`atMost(180, TimeUnit.SECONDS)`) 
for a replacement Pod to be `Running`, for the job to be back in `RUNNING`, and 
for the promoted master to be visible to the probe. Both happened on JDK 17, 
while JDK 11 passed on this head, and both JDKs passed on `4504ff81` / 
`713de07d`. The PR's own diff has not changed since those heads (this head only 
merges `dev`), so this is still not a source regression, but it is a recurring 
timing problem in this PR's new HA test, not a one-off.
   - `dev` has nothing that would change this (the test is new in this PR), so 
syncing again will not help.
   
   Because a new E2E that fails 2 of the last 4 JDK 17 attempts would turn into 
a flaky lane for everyone after merge, I'd treat this as the one remaining item 
before merge. Two small options:
   
   - **Option A (smallest):** give the active-master recovery waits their own, 
longer budget (for example 300 seconds, shared through one constant), and log 
the replacement Pod phase plus the probe result on each failed poll so the next 
failure shows which of the three conditions was still false.
   - **Option B:** split the wait into stages (replacement Pod `Running` -> job 
back to `RUNNING` -> promoted master visible), each with its own timeout and a 
clear failure message. Slightly more code, but a future failure points straight 
at the slow stage.
   
   Everything else from my last review is unchanged: no source blocker from my 
side, and @nzw921rx's change request still needs to be completed by a 
write-access maintainer. Happy to re-check as soon as you push the update!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to