goutamadwant opened a new pull request, #13245: URL: https://github.com/apache/seatunnel/pull/13245
### Purpose of this pull request Closes #13244. `HbaseSinkWriter` wrote every row as a `Put`, so a CDC `DELETE` row put its old values back into the table, and `prepareCommit()` did nothing while the client's `BufferedMutator` (`write_buffer_size`, 8 MB by default) was flushed only on `close()`, so a checkpoint could complete with rows still in memory. - `write()` now routes by `RowKind`: `INSERT` and `UPDATE_AFTER` stay a `Put`; `DELETE` becomes a `Delete` of the column families this writer maps (other families of a shared table are left alone), with the same timestamp rule as the `Put` (`version_column` when set, otherwise the current time) and the same `wal_write` durability; `UPDATE_BEFORE` is not written on its own, because the following `UPDATE_AFTER` overwrites the same row key and a tombstone written in the same millisecond would hide it. When the `UPDATE_AFTER` carries another row key, the old row is deleted first. - `prepareCommit()` flushes the mutator, so a checkpoint only completes once its rows have been accepted by the region servers. The HBase client drops the mutations of a failed flush, so after one the writer refuses further writes and flushes instead of letting a table-level retry acknowledge the checkpoint with those rows missing. - A null or non-`BIGINT` `version_column` value fails with a message naming the column instead of an NPE. - `HbaseClient.mutate` takes a `Mutation`, `flush()` is exposed, and `close()` releases the admin and connection even when the final flush fails. - Known limitation, documented: without `version_column`, a `DELETE` and an `INSERT` of one row key in the same millisecond resolve to the delete (HBase tombstone semantics); the docs recommend `version_column` for CDC sources. - Docs (EN/ZH): `cdc` key feature and two notes on delete handling and checkpoint flush. ### Does this PR introduce _any_ user-facing change? Yes. A `DELETE` row now removes the mapped column families of the HBase row instead of rewriting its old values, an update that moves a row-key column removes the old row, and `UPDATE_BEFORE` rows are no longer written on their own. Jobs that relied on deletes being ignored will see rows disappear, which is what the source said. ### How was this patch tested? - `HbaseSinkWriterTest` (12 cases): delete written as a `Delete` of the mapped families only, update keeping the key writes one `Put`, update moving the key deletes the old row first, unpaired update-before dropped, `version_column` is the timestamp of `Put` and `Delete`, null version rejected with the column name, delete follows `wal_write`, `prepareCommit` flushes, writer unusable after a failed flush, insert/update-after still a `Put`. Four of them fail on dev and pass with the fix. - `HbaseClientTest`: `mutate` buffers `Put` and `Delete`, `flush` delegates to the mutator. - Full `connector-hbase` unit suite (56 tests) on JDK 11 and JDK 17. - New E2E `HbaseIT#testHbaseSinkWithCdcRows` (FakeSource with insert, in-place update, key-moving update and delete rows) passes on the Spark 3.5 container locally (`HbaseIT` is disabled on Zeta upstream, so the suite runs on Flink/Spark); the existing `testHbaseSink` passes alongside it. ### Check list * [x] No new dependency * [x] Documentation updated (`docs/en|zh/connectors/sink/Hbase.md`) * [x] `incompatible-changes.md` not needed * [x] Not a connector addition -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
