SEZ9 commented on PR #13060:
URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6842064784

   Thanks for the question, @nzw921rx. Capping parallelism at 1 for 
EXACTLY_ONCE would indeed sidestep most of the rescale-related concerns here 
(the Zeta-vs-Flink state redistribution mismatch, mixed random prefixes when 
`transaction_prefix` is unset, and the concurrent legacy-namespace probing), 
simply because there would be nothing to redistribute. But I don't think it's 
the right trade-off as the primary fix: users already run EXACTLY_ONCE Kafka 
sinks at parallelism > 1 today, and turning that into a hard validation error 
would be a functional regression rather than a correctness fix. It also 
wouldn't address the Spark taskId / retry fencing gap, which exists even at 
parallelism 1.
   
   Where I'd be open to a validation-style guard is as a narrow safety net on 
top of the per-writer namespace approach, not as a replacement for it — for 
example rejecting restores only in the specific cases we know we can't reason 
about safely, rather than rejecting parallelism > 1 altogether.
   
   Concrete asks that remain on the current scope:
   
   1. **Flink restore (PR13060-F1)**: please confirm how 
`validateRestoredStateOwner` in `KafkaSinkWriter` behaves when Flink hands a 
scale-down writer a contiguous chunk of states instead of the Zeta modulo 
assignment. If the engine-specific redistribution can't be detected, the check 
should degrade to a warning rather than throwing `IllegalStateException`.
   2. **Unset `transaction_prefix` (PR13060-F2)**: the mixed-prefix rejection 
in `restoreState()` needs a path for writers that each generated a random 
prefix; otherwise every scale-down restore without an explicit prefix fails.
   3. **No-state writer sequence start (PR13060-F3)**: a scale-up writer should 
not start at sequence 1 independent of the job's checkpoint id; please align it 
with the restored checkpoint id or explain why fencing is still guaranteed.
   4. **Spark taskId as subtask index (PR13060-F4)**: please state how the 
previous attempt's open transaction is fenced when the subtask index changes on 
retry.
   5. **Docs (PR13060-F7)**: align the `transaction_prefix` option row in 
`docs/en/connectors/sink/Kafka.md` with the new ID shape, and scope the 
rescaling claim to the engines where it is actually verified.
   
   The legacy-prefix probing cost (PR13060-F5/F6) has been treated as a 
deferred follow-up; I'm fine keeping that separate from the correctness items 
above.
   
   <!-- streview-comment:2308 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to