DanielLeens commented on issue #13298: URL: https://github.com/apache/seatunnel/issues/13298#issuecomment-6949673631
Classification: F / Connector-V2 Kafka legacy transactional-recovery cleanup. Thanks for restoring this follow-up and for keeping it separate from #13060. I rechecked remote `dev` at `50cf6b6711f9316557daa3a5e17defdab4247953` and the current #13060 head `7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5`. On `dev`, `KafkaSinkWriter` starts recovery cleanup from its locally assigned `KafkaSinkState` list and asks `KafkaTransactionSender` to probe `<prefix>-<sequence>` IDs until it initializes the first unused ID. #13060 is still open; its current design deliberately keeps that legacy shared-prefix cleanup per writer and explicitly defers F5/F6 cross-writer coordination. So this tracker is valid, but it must remain a follow-up rather than being folded into or implemented in parallel with #13060. Please add a short design note here before opening a code PR, after #13060 has merged or has otherwise established its final recovery contract. In particular, it needs to define: 1. the persisted, engine-neutral input that determines cleanup coverage and the per-prefix owner; a writer-local list or current-index/modulo rule cannot prove all legacy prefixes are covered; 2. owner failure, retry, and restore behavior, including how a replacement can safely resume cleanup without leaving a sibling namespace fenced or an unfinished legacy transaction visible; 3. mixed historical prefixes and the actual Flink and Zeta redistribution cases, rather than assuming one assignment scheme; 4. a termination and complexity argument that reduces redundant producer initialization without an arbitrary scan cap that could skip an unfinished transaction; and 5. an executable proof matrix: at least two restoring writers, shared and mixed prefixes, owner failure/retry, `read_committed` visibility for abandoned versus new records, and recorded producer-initialization/recovery-time measurements. Keep the existing per-writer cleanup until that replacement proves equivalent fencing coverage. The documented no-retained-state `transaction.timeout.ms` boundary is separate and should not be broadened by this optimization. Once the design has maintainer agreement, a focused Connector-V2 PR with the corresponding recovery tests will be the right next step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
