SEPURI-SAI-KRISHNA opened a new issue, #13286:
URL: https://github.com/apache/seatunnel/issues/13286

   ### Search before asking
   
   - [X] I had searched in the 
[issues](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Bug%22)
 and found no similar issues.
   
   ### What happened
   
   The Cassandra sink can lose rows while the job finishes successfully, with 
either value of `async_write`. I ran a FakeSource to Cassandra batch job on 
Zeta (`dev`) against a real Cassandra 4.1.1, driver 4.14.0, with 3 rows where 
row 2 has a 17 MiB text value, above the 16 MiB mutation limit of Cassandra 
4.1, so the server rejects it:
   
   | `async_write` | Job status | ids in the table |
   | --- | --- | --- |
   | `true` (the default) | FINISHED | 1, 3 |
   | `false` | FINISHED | 1 |
   
   There are two causes, and a smaller third problem.
   
   **1. A failed async write is only logged.** With `async_write = true`, 
`CassandraSinkWriter.write()` sends each row with `session.executeAsync`, and 
`flush()` only attaches a callback that logs a failure with `log.error`. 
Nothing waits for these writes or rethrows a failure, so a write that the 
server rejects, or that times out, is lost. The job above logs one `ERROR` line 
(below) and finishes without row 2.
   
   **2. A checkpoint does not flush.** `CassandraSinkWriter` extends 
`AbstractSinkWriter`, whose `prepareCommit()` returns `Optional.empty()`, and 
does not override `snapshotState()`. At a checkpoint the writer neither waits 
for the async writes in flight nor sends the rows buffered for the next 
synchronous batch (up to `batch_size`, 5000 by default). The checkpoint 
completes, so the source position moves past rows that are not in Cassandra yet.
   
   The `async_write = false` job above shows the result. Its final checkpoint 
completes while all 3 rows are still in the batch buffer. `close()` then sends 
them: the batch fails, the one-by-one retry writes row 1 and throws on row 2, 
and the task fails. Zeta restores the pipeline from the completed checkpoint 
(`Restore time 1`), the source has nothing left to read, and the job finishes. 
Rows 2 and 3 are lost.
   
   Driving the `dev` writer directly, the way the engine does at a checkpoint, 
with `async_write = false`:
   
   ```
   write ids 1, 2, 3
   prepareCommit() returned Optional.empty
   snapshotState(1) returned []
   ids in the table after the checkpoint, before close(): []
   ids in the table after close(): [1, 2, 3]
   ```
   
   **3. Smaller: `close()` leaves the session open when the flush fails.** 
`close()` calls `flush()` and closes the session only after it, so a failing 
flush skips `session.close()`. When I drove the writer directly, the process 
did not exit after the synchronous writer failed and was ended by a 300 s 
timeout; that writer's session was the only one left open.
   
   #13126 (for #13124) rejects a NULL primary key before the write, which 
removes one cause of a rejected write. This issue covers every other failure, 
as noted there.
   
   Proposed fix, in one PR, with no option or default change:
   
   - keep the async writes in flight, and wait for them when `batch_size` is 
reached, at a checkpoint (`prepareCommit`) and in `close()`, failing the writer 
with the first error (later ones added as suppressed);
   - at a checkpoint, also send the rows buffered for the synchronous batch, so 
a failure there fails the checkpoint instead of surfacing after it;
   - close the session in `close()` also when the flush fails;
   - `CassandraIT` cases for a rejected row with `async_write` on and off that 
expect the job to fail, and unit tests for the checkpoint flush.
   
   A job that loses rows this way today will fail instead, so I would also add 
an `incompatible-changes.md` entry.
   
   I am willing to submit the PR.
   
   ### SeaTunnel Version
   
   dev (66c9b4b762). 2.3.12, 2.3.13 and 3.0.0 have the same code.
   
   ### SeaTunnel Config
   
   ```conf
   env {
     parallelism = 1
     job.mode = "BATCH"
   }
   source {
     FakeSource {
       plugin_output = "fake"
       schema = { fields { id = int, v = string } }
       rows = [
         { kind = INSERT, fields = [1, "small"] },
         # the second value is 17 MiB of "x"
         { kind = INSERT, fields = [2, "xxx..."] },
         { kind = INSERT, fields = [3, "small"] }
       ]
     }
   }
   sink {
     Cassandra {
       plugin_input = "fake"
       host = "127.0.0.1:9042"
       datacenter = "datacenter1"
       keyspace = "repro"
       table = "t"
       async_write = true   # and false for the second run
     }
   }
   ```
   
   The table is `CREATE TABLE repro.t (id int PRIMARY KEY, v text)`.
   
   ### Running Command
   
   I ran the jobs on an embedded Zeta server from a test harness, as 
`SeaTunnelClientTest` does, and drove the writer directly for the checkpoint 
and session cases. With a binary distribution it is:
   
   ```shell
   bin/seatunnel.sh --config job.conf -m local
   ```
   
   ### Error Exception
   
   ```log
   async_write = true: no exception reaches the job; the writer only logs
   ERROR 
org.apache.seatunnel.connectors.seatunnel.cassandra.sink.CassandraSinkWriter - 
com.datastax.oss.driver.api.core.servererrors.InvalidQueryException: Rejected 
an oversized mutation (17825848/16777216) for keyspace: repro. Top keys are: 
job_async_true.2
   
   async_write = false: the task fails in CassandraSinkWriter.close(), and the 
restored pipeline finishes
   SubPlan - Job job_async_false (...), Pipeline: [(1/1)] turned from state 
FAILING to FAILED.
   SubPlan - Restore time 1, pipeline Job job_async_false (...), Pipeline: 
[(1/1)]
   SubPlan - Job job_async_false (...), Pipeline: [(1/1)] turned from state 
RUNNING to FINISHED.
   ```
   
   ### Zeta or Flink or Spark Version
   
   Zeta (dev). The writer code is the same on every engine.
   
   ### Java or Scala Version
   
   JDK 17
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to