zephyr45 commented on PR #13060:
URL: https://github.com/apache/seatunnel/pull/13060#issuecomment-6899295671

   @SEZ9 I rechecked your latest questions against `7c272c5a` and reran the 
affected tests locally.
   
   - **F1/F2:** 
[Restore](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L243-L296)
 consumes the engine-assigned states; `validateRestoredStateOwner` and the 
modulo rejection are gone. 
[Parsing](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L362-L427)
 uses each state's own saved prefix, so historical mixed prefixes are accepted. 
The contiguous-assignment, mixed-prefix and repeated `3 → 4 → 3 → 4 → 3` 
regressions passed.
   - **F4:** Kafka [uses the stable 
index](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L103-L106),
 which is Spark's partition ID. Other sinks retain the task-attempt index. The 
[broker retry 
test](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/java/org/apache/seatunnel/e2e/connector/kafka/KafkaIT.java#L2399-L2460)
 passed with task IDs 99 and 100 sharing stable index 0: the abandoned record 
stayed invisible and `read_committed` progress resumed.
   - **F7:** The [option row and 
semantics](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/docs/en/connectors/sink/Kafka.md#L66-L108)
 describe the ID shape, Zeta/Flink stateful recovery and Spark's speculation 
restriction; ZH matches.
   
   **F3 still has a limit:** a writer with no retained state opens sequence 1, 
fencing its previous holder. Later abandoned transactions absent from 
checkpoint state can remain open until `transaction.timeout.ms`. The 
[fresh-writer 
path](https://github.com/apache/seatunnel/blob/7c272c5a246e617bff1f737fa4e78ed3a1ab7bf5/seatunnel-connectors-v2/connector-kafka/src/main/java/org/apache/seatunnel/connectors/seatunnel/kafka/sink/KafkaSinkWriter.java#L319-L350)
 deliberately avoids scanning all history, which caused growing startup work 
across Spark micro-batches. Aligning with a checkpoint counter would not 
identify those missing transactions; Flink's adapter counters can also differ 
between writers.
   
   The Java 11 rerun passed 583 tests, including the three broker regressions, 
parallel Kafka batch jobs across Flink/Spark/Zeta, and Zeta's 3→2 restore. Real 
Spark 2.4 and 3.3 Doris 2PC tests also passed both batches; the old factories 
failed on batch two with `Label Already Exists`. Flink rescaling remains 
covered by model/adapter tests, not a real Flink rescale run.
   
   Is that documented F3 timeout boundary acceptable for this PR? If so, could 
you finish your re-review of this head? F5/F6 remains the agreed optimization 
follow-up.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to