SEPURI-SAI-KRISHNA commented on PR #13288:
URL: https://github.com/apache/seatunnel/pull/13288#issuecomment-6943516856

   @DanielLeens Thanks for the review. All three issues are addressed in 
`b1e7f560b3`:
   
   **Issue 1, Spark abort.** `CassandraSinkWriter` now implements 
`abortPrepare()`: it drops the buffered rows and the writes in flight and 
closes the session, without sending anything, and never goes through the flush 
of `close()`. The engine writes those rows again when it retries or restores. 
Zeta also calls it when a checkpoint fails; the `close()` that follows writes 
nothing and does not close the session again. Because `AbstractSinkWriter` 
declares `abortPrepare()` final, the writer implements `SinkWriter` and the 
sink implements `SeaTunnelSink` directly, as Neo4j does. The Spark exclusion is 
removed from `testNullPrimaryKeyFailsTheJob` and `testRejectedRowFailsTheJob`. 
Locally both jobs now fail on Spark 2.4.6 and 3.3.0 in under a minute; with the 
writer this PR was opened with (`c7134db509`) the first one hung on Spark 2.4.6 
until cut at 25 minutes. This fixes the session for Cassandra only; the general 
Spark adapter change stays in #13156.
   
   **Issue 2, batch limit.** The count is now taken after the row 
(`incrementAndGet() >= batch_size`), so at most `batch_size` rows are buffered 
or in flight. `batchSizeOfOneWaitsForEachAsyncWrite` asserts that the first 
write waits with `batch_size = 1`, `batchIsSentWhenItReachesTheBatchSize` 
covers the synchronous batch, and the old count fails both and 
`failedAsyncWriteFailsTheWriteThatReachesTheBatchSize`. The EN and ZH 
`batch_size` text describes both modes.
   
   **Issue 3, metadata.** I kept the fail-closed check from #13126 (your Issue 
1 there). It names the table and column before anything is sent. Without it, a 
synchronous batch with a NULL key fails as a batch, and the one-by-one retry 
writes the rows before the NULL key and then fails. The writer already reads 
the schema from the server once more before it gives up. The sink pages (EN and 
ZH) now document it, with `CASSANDRA-06`, as you suggested.
   
   The threshold comment matches the code, and every write failure names the 
table. New abort tests (no send, one close after a failed checkpoint, no wait 
on writes in flight, failing close) each catch a deliberate break. 
`connector-cassandra`: 56 tests pass. Whole `CassandraIT` locally: Zeta 12, 
Flink 24 and Spark 16 tests pass (`testDryRunPackagedCli` stays skipped on 
Flink and Spark).
   
   Stack order: once #13126 merges I rebase this onto `dev`, so only its own 
commits remain.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to