DanielLeens commented on issue #11735: URL: https://github.com/apache/seatunnel/issues/11735#issuecomment-5239948873
Thanks for formalizing the STIP. I re-checked the current `dev` engine code before replying here, and the problem statement is grounded: - `PhysicalPlan` still keeps only the first pipeline error via `errorBySubPlan.compareAndSet(null, ...)`. - pipeline- and task-group-level `stateTimestamps` are already recorded internally. - the existing public timestamp read path exposed by `JobMaster` is still job-level. So the direction in this STIP is reasonable: there is already enough raw runtime state to build a bounded failure-history contract, but that contract does not exist yet as a stable read path. For v1, I would recommend keeping the stable boundary slightly narrow: - pipeline-attempt grouping is a good primary model; - task / worker attribution should remain optional exactly as you described, because current transport paths do not always provide it reliably; - keep the existing `errorMessage` contract intact for compatibility; - make failure-history persistence best-effort so it cannot interfere with the original fail / restore path. The main review points I would focus on next in this STIP / #11734 are: - the exact dedupe key and how it behaves across repeated delivery of the same terminal state; - truncation metadata and concrete size limits for message / stack trace fields; - how the bounded history snapshot is persisted for finished-job backends without turning this into an unbounded storage contract. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
