SEZ9 commented on PR #13060: URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6842064784
Thanks for the question, @nzw921rx. Capping parallelism at 1 for EXACTLY_ONCE would indeed sidestep most of the rescale-related concerns here (the Zeta-vs-Flink state redistribution mismatch, mixed random prefixes when `transaction_prefix` is unset, and the concurrent legacy-namespace probing), simply because there would be nothing to redistribute. But I don't think it's the right trade-off as the primary fix: users already run EXACTLY_ONCE Kafka sinks at parallelism > 1 today, and turning that into a hard validation error would be a functional regression rather than a correctness fix. It also wouldn't address the Spark taskId / retry fencing gap, which exists even at parallelism 1. Where I'd be open to a validation-style guard is as a narrow safety net on top of the per-writer namespace approach, not as a replacement for it — for example rejecting restores only in the specific cases we know we can't reason about safely, rather than rejecting parallelism > 1 altogether. Concrete asks that remain on the current scope: 1. **Flink restore (PR13060-F1)**: please confirm how `validateRestoredStateOwner` in `KafkaSinkWriter` behaves when Flink hands a scale-down writer a contiguous chunk of states instead of the Zeta modulo assignment. If the engine-specific redistribution can't be detected, the check should degrade to a warning rather than throwing `IllegalStateException`. 2. **Unset `transaction_prefix` (PR13060-F2)**: the mixed-prefix rejection in `restoreState()` needs a path for writers that each generated a random prefix; otherwise every scale-down restore without an explicit prefix fails. 3. **No-state writer sequence start (PR13060-F3)**: a scale-up writer should not start at sequence 1 independent of the job's checkpoint id; please align it with the restored checkpoint id or explain why fencing is still guaranteed. 4. **Spark taskId as subtask index (PR13060-F4)**: please state how the previous attempt's open transaction is fenced when the subtask index changes on retry. 5. **Docs (PR13060-F7)**: align the `transaction_prefix` option row in `docs/en/connectors/sink/Kafka.md` with the new ID shape, and scope the rescaling claim to the engines where it is actually verified. The legacy-prefix probing cost (PR13060-F5/F6) has been treated as a deferred follow-up; I'm fine keeping that separate from the correctness items above. <!-- streview-comment:2308 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
