SEPURI-SAI-KRISHNA opened a new issue, #13286: URL: https://github.com/apache/seatunnel/issues/13286
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22) and found no similar issues. ### What happened The Cassandra sink can lose rows while the job finishes successfully, with either value of `async_write`. I ran a FakeSource to Cassandra batch job on Zeta (`dev`) against a real Cassandra 4.1.1, driver 4.14.0, with 3 rows where row 2 has a 17 MiB text value, above the 16 MiB mutation limit of Cassandra 4.1, so the server rejects it: | `async_write` | Job status | ids in the table | | --- | --- | --- | | `true` (the default) | FINISHED | 1, 3 | | `false` | FINISHED | 1 | There are two causes, and a smaller third problem. **1. A failed async write is only logged.** With `async_write = true`, `CassandraSinkWriter.write()` sends each row with `session.executeAsync`, and `flush()` only attaches a callback that logs a failure with `log.error`. Nothing waits for these writes or rethrows a failure, so a write that the server rejects, or that times out, is lost. The job above logs one `ERROR` line (below) and finishes without row 2. **2. A checkpoint does not flush.** `CassandraSinkWriter` extends `AbstractSinkWriter`, whose `prepareCommit()` returns `Optional.empty()`, and does not override `snapshotState()`. At a checkpoint the writer neither waits for the async writes in flight nor sends the rows buffered for the next synchronous batch (up to `batch_size`, 5000 by default). The checkpoint completes, so the source position moves past rows that are not in Cassandra yet. The `async_write = false` job above shows the result. Its final checkpoint completes while all 3 rows are still in the batch buffer. `close()` then sends them: the batch fails, the one-by-one retry writes row 1 and throws on row 2, and the task fails. Zeta restores the pipeline from the completed checkpoint (`Restore time 1`), the source has nothing left to read, and the job finishes. Rows 2 and 3 are lost. Driving the `dev` writer directly, the way the engine does at a checkpoint, with `async_write = false`: ``` write ids 1, 2, 3 prepareCommit() returned Optional.empty snapshotState(1) returned [] ids in the table after the checkpoint, before close(): [] ids in the table after close(): [1, 2, 3] ``` **3. Smaller: `close()` leaves the session open when the flush fails.** `close()` calls `flush()` and closes the session only after it, so a failing flush skips `session.close()`. When I drove the writer directly, the process did not exit after the synchronous writer failed and was ended by a 300 s timeout; that writer's session was the only one left open. #13126 (for #13124) rejects a NULL primary key before the write, which removes one cause of a rejected write. This issue covers every other failure, as noted there. Proposed fix, in one PR, with no option or default change: - keep the async writes in flight, and wait for them when `batch_size` is reached, at a checkpoint (`prepareCommit`) and in `close()`, failing the writer with the first error (later ones added as suppressed); - at a checkpoint, also send the rows buffered for the synchronous batch, so a failure there fails the checkpoint instead of surfacing after it; - close the session in `close()` also when the flush fails; - `CassandraIT` cases for a rejected row with `async_write` on and off that expect the job to fail, and unit tests for the checkpoint flush. A job that loses rows this way today will fail instead, so I would also add an `incompatible-changes.md` entry. I am willing to submit the PR. ### SeaTunnel Version dev (66c9b4b762). 2.3.12, 2.3.13 and 3.0.0 have the same code. ### SeaTunnel Config ```conf env { parallelism = 1 job.mode = "BATCH" } source { FakeSource { plugin_output = "fake" schema = { fields { id = int, v = string } } rows = [ { kind = INSERT, fields = [1, "small"] }, # the second value is 17 MiB of "x" { kind = INSERT, fields = [2, "xxx..."] }, { kind = INSERT, fields = [3, "small"] } ] } } sink { Cassandra { plugin_input = "fake" host = "127.0.0.1:9042" datacenter = "datacenter1" keyspace = "repro" table = "t" async_write = true # and false for the second run } } ``` The table is `CREATE TABLE repro.t (id int PRIMARY KEY, v text)`. ### Running Command I ran the jobs on an embedded Zeta server from a test harness, as `SeaTunnelClientTest` does, and drove the writer directly for the checkpoint and session cases. With a binary distribution it is: ```shell bin/seatunnel.sh --config job.conf -m local ``` ### Error Exception ```log async_write = true: no exception reaches the job; the writer only logs ERROR org.apache.seatunnel.connectors.seatunnel.cassandra.sink.CassandraSinkWriter - com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: Rejected an oversized mutation (17825848/16777216) for keyspace: repro. Top keys are: job_async_true.2 async_write = false: the task fails in CassandraSinkWriter.close(), and the restored pipeline finishes SubPlan - Job job_async_false (...), Pipeline: [(1/1)] turned from state FAILING to FAILED. SubPlan - Restore time 1, pipeline Job job_async_false (...), Pipeline: [(1/1)] SubPlan - Job job_async_false (...), Pipeline: [(1/1)] turned from state RUNNING to FINISHED. ``` ### Zeta or Flink or Spark Version Zeta (dev). The writer code is the same on every engine. ### Java or Scala Version JDK 17 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
