zhangshenghang opened a new issue, #13323:
URL: https://github.com/apache/seatunnel/issues/13323

   ### What happened
   
   `pipelineRestoreNum` is memory-only, so a master failover resets the 
`job.retry.times` budget: a job that should exhaust its restarts can restart 
indefinitely across failovers (dev @ 89f91a89af).
   
   Chain:
   
   1. `SubPlan` initializes `pipelineRestoreNum = new AtomicInteger()` in the 
constructor (`SubPlan.java:103,164`); the constructor restores the pipeline 
*status* from `runningJobStateIMap` but never the restore count.
   2. On master failover, `CoordinatorService` rebuilds every 
JobMaster/PhysicalPlan/SubPlan from scratch; every SubPlan starts again from 
zero restores.
   3. `canRestorePipeline` (`SubPlan.java:417-419`) only checks 
`jobMaster.isNeedRestore() && getPipelineRestoreNum() < pipelineMaxRestoreNum` 
— with the counter reset, a crash-looping job gets a fresh budget on every 
master switchover, so `job.retry.times` no longer bounds total restarts.
   
   The same applies to failure-cause history: `errorByPhysicalVertex` and the 
newly added `restoreCauses` (#13154) are memory-only and are lost on failover, 
which defeats their purpose exactly when a job is crash-looping through master 
switches.
   
   ### Impact
   
   `job.retry.times` cannot cap a crash loop that coincides with master 
failovers; jobs may restart forever. Restore-cause history vanishes at the 
moment it is most needed.
   
   ### Suggested fix
   
   Persist the per-pipeline restore counter (e.g. alongside the pipeline state 
in the job state IMap) and restore it when rebuilding SubPlans after failover; 
persist `restoreCauses` similarly.
   
   ### SeaTunnel Version
   
   dev (89f91a89af), Zeta engine
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to