SEPURI-SAI-KRISHNA opened a new pull request, #13288:
URL: https://github.com/apache/seatunnel/pull/13288
Yes, glad to contribute to Apache SeaTunnel.
### Purpose of this pull request
Closes #13286.
Stacked on #13126, which changes the same writer and adds
`CassandraSinkWriterTest`: until #13126 merges, this PR also shows its commits,
and only the last commit belongs here. I will rebase once #13126 is in.
The Cassandra sink could finish a job while losing rows, with either value
of `async_write`:
- with `async_write = true`, the default, a write that Cassandra rejected or
that timed out was only logged;
- the writer did not flush at a checkpoint (`prepareCommit` came from
`AbstractSinkWriter` and did nothing). With `async_write = false` the rows
buffered for the next batch were sent only after the final checkpoint of a
batch job, so when Cassandra rejected one, Zeta restored the pipeline from that
checkpoint, the source had nothing left, and the job finished without the rows.
#13286 has both jobs and their logs.
Changes in `CassandraSinkWriter`:
- `prepareCommit()` flushes, as the Redis and Google Pub/Sub sinks do. Zeta
calls it at every snapshot barrier, Flink before the barrier and Spark at
commit.
- The flush, after `batch_size` rows, at a checkpoint and in `close()`: with
`async_write = true` it waits for every write in flight and fails with
`CASSANDRA-07`, for example `1 of 2 writes to table t failed; the first failure
is the cause`, with up to 9 more failures as suppressed. With `async_write =
false` it sends the buffered batch as before (one batch, then one by one if the
batch fails); a row that still fails throws with the table name, and an empty
batch is not sent.
- `close()` closes the session also when the last flush fails. Before, a
failed flush left it open; in my writer-level run the process then did not exit.
- The comments of #13126 that said a rejected async write is only logged now
give the remaining reason for the NULL key check: it names the table and column
before anything is sent.
Notes for reviewers:
- Waiting at `batch_size` also bounds the async writes in flight to
`batch_size`; before, nothing waited. The rows are still sent one by one with
`executeAsync`.
- The sink is at-least-once: a job restored from a checkpoint writes the
rows since that checkpoint again, which an `INSERT` by primary key absorbs.
- #13126 fails the writer when it cannot read the primary key, because a
rejected NULL key was only logged with `async_write`. With this PR such a write
fails the job anyway, so that check could become a warning; I left it as
reviewed in #13126.
- #12993 edits the same lines of the Cassandra sink pages (`async_write`,
notes); whichever lands second rebases.
### Does this PR introduce _any_ user-facing change?
Yes. A row that the Cassandra sink cannot write now fails the job with
`CASSANDRA-07`, the table and the error of Cassandra, with either value of
`async_write`; before, the job could finish without the row. No option or
default changes. The EN and ZH `incompatible-changes.md` have an entry, and the
sink pages describe the checkpoint behavior of `async_write` and `batch_size`.
### How was this patch tested?
- `CassandraSinkWriterTest`, with a mocked session (7 new tests): a failed
async write fails the checkpoint, every failure is reported (count, first as
cause, the next as suppressed), a failed write fails the write that reaches
`batch_size`, a checkpoint passes when every write succeeded, the synchronous
batch is sent at the checkpoint and an empty one is not, a rejected row of the
batch fails the checkpoint, and `close()` fails and still closes the session.
`connector-cassandra`: 50 tests pass.
- I broke the change on purpose, one mutation at a time, and a test failed
each time: no flush in `prepareCommit` (4 tests fail), async failures ignored
(4), the session not closed after a failed flush (1), an empty batch sent (1),
no wait at `batch_size` (1).
- `CassandraIT#testRejectedRowFailsTheJob`: a FakeSource row with a 17 MiB
text value, which Cassandra 4.1 rejects (`Rejected an oversized mutation`),
with `async_write = true` and `false` and the default restarts. Each job must
fail with the error above and leave no row. Spark is disabled as for
`testNullPrimaryKeyFailsTheJob` (#13155).
- Local runs of the whole `CassandraIT`: Zeta 12 tests pass; Flink 1.13.6,
1.15.3, 1.18.0 and 1.20.1, 24 run, 0 failures (1 skipped:
`testDryRunPackagedCli` is disabled on Flink). With the writer of #13126 the
new test fails: `async_write = false ==> expected: not equal but was: <0>`, the
job finished.
- `spotless:check`, `verify -DskipTests` of both modules, and `MarkdownTest`.
### Check list
* [x] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
(no new jar)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated: (not a new connector)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]