SEZ9 commented on issue #13286:
URL: https://github.com/apache/seatunnel/issues/13286#issuecomment-6936967256

   Thanks for the detailed report and reproduction (Zeta on `dev`, Cassandra 
4.1.1, driver 4.14.0, a 17 MiB row above the 16 MiB mutation limit). A job that 
ends as FINISHED with only ids 1, 3 (`async_write = true`) or only id 1 
(`async_write = false`) in the table is silent data loss, so this is worth 
fixing.
   
   I have not verified the `CassandraSinkWriter` analysis against `dev` myself, 
so the three causes still need to be confirmed during review. The direction you 
propose sounds reasonable to me:
   
   - wait for in-flight async writes when `batch_size` is reached, in 
`prepareCommit()` and in `close()`, and fail the writer on the first error;
   - send the buffered synchronous batch at a checkpoint, so a failure fails 
the checkpoint instead of surfacing after it;
   - close the session in `close()` even when the flush fails.
   
   The `CassandraIT` cases for a rejected row with `async_write` on and off, 
the unit tests for the checkpoint flush, and the `incompatible-changes.md` 
entry would all be good to include, since jobs that lose rows today will fail 
instead.
   
   Since you are willing to submit the PR, please go ahead, and we can discuss 
the details in review.
   
   <!-- streview-comment:2396 -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to