goutamadwant opened a new pull request, #13245:
URL: https://github.com/apache/seatunnel/pull/13245

   ### Purpose of this pull request
   
   Closes #13244.
   
   `HbaseSinkWriter` wrote every row as a `Put`, so a CDC `DELETE` row put its 
old values back into the table, and `prepareCommit()` did nothing while the 
client's `BufferedMutator` (`write_buffer_size`, 8 MB by default) was flushed 
only on `close()`, so a checkpoint could complete with rows still in memory.
   
   - `write()` now routes by `RowKind`: `INSERT` and `UPDATE_AFTER` stay a 
`Put`; `DELETE` becomes a `Delete` of the column families this writer maps 
(other families of a shared table are left alone), with the same timestamp rule 
as the `Put` (`version_column` when set, otherwise the current time) and the 
same `wal_write` durability; `UPDATE_BEFORE` is not written on its own, because 
the following `UPDATE_AFTER` overwrites the same row key and a tombstone 
written in the same millisecond would hide it. When the `UPDATE_AFTER` carries 
another row key, the old row is deleted first.
   - `prepareCommit()` flushes the mutator, so a checkpoint only completes once 
its rows have been accepted by the region servers. The HBase client drops the 
mutations of a failed flush, so after one the writer refuses further writes and 
flushes instead of letting a table-level retry acknowledge the checkpoint with 
those rows missing.
   - A null or non-`BIGINT` `version_column` value fails with a message naming 
the column instead of an NPE.
   - `HbaseClient.mutate` takes a `Mutation`, `flush()` is exposed, and 
`close()` releases the admin and connection even when the final flush fails.
   - Known limitation, documented: without `version_column`, a `DELETE` and an 
`INSERT` of one row key in the same millisecond resolve to the delete (HBase 
tombstone semantics); the docs recommend `version_column` for CDC sources.
   - Docs (EN/ZH): `cdc` key feature and two notes on delete handling and 
checkpoint flush.
   
   ### Does this PR introduce _any_ user-facing change?
   
   Yes. A `DELETE` row now removes the mapped column families of the HBase row 
instead of rewriting its old values, an update that moves a row-key column 
removes the old row, and `UPDATE_BEFORE` rows are no longer written on their 
own. Jobs that relied on deletes being ignored will see rows disappear, which 
is what the source said.
   
   ### How was this patch tested?
   
   - `HbaseSinkWriterTest` (12 cases): delete written as a `Delete` of the 
mapped families only, update keeping the key writes one `Put`, update moving 
the key deletes the old row first, unpaired update-before dropped, 
`version_column` is the timestamp of `Put` and `Delete`, null version rejected 
with the column name, delete follows `wal_write`, `prepareCommit` flushes, 
writer unusable after a failed flush, insert/update-after still a `Put`. Four 
of them fail on dev and pass with the fix.
   - `HbaseClientTest`: `mutate` buffers `Put` and `Delete`, `flush` delegates 
to the mutator.
   - Full `connector-hbase` unit suite (56 tests) on JDK 11 and JDK 17.
   - New E2E `HbaseIT#testHbaseSinkWithCdcRows` (FakeSource with insert, 
in-place update, key-moving update and delete rows) passes on the Spark 3.5 
container locally (`HbaseIT` is disabled on Zeta upstream, so the suite runs on 
Flink/Spark); the existing `testHbaseSink` passes alongside it.
   
   ### Check list
   
   * [x] No new dependency
   * [x] Documentation updated (`docs/en|zh/connectors/sink/Hbase.md`)
   * [x] `incompatible-changes.md` not needed
   * [x] Not a connector addition
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to