goutamadwant opened a new issue, #13244:
URL: https://github.com/apache/seatunnel/issues/13244

   ### Search before asking
   
   - [x] I had searched in the issues and found no similar issues.
   
   ### What happened
   
   Two defects in `HbaseSinkWriter`, both on dev (`9cd7defe84`) and in 3.1.0:
   
   1. **Deleted rows come back.** `write()` turns every row into a `Put` 
(`HbaseSinkWriter.java:104-107`); there is no `RowKind` check anywhere in the 
connector. A CDC `DELETE` row still carries the old column values, so the sink 
writes them back as a normal row. An `UPDATE_BEFORE` row is written the same 
way and is then overwritten by the `UPDATE_AFTER` for the same key, which hides 
the problem for updates but not for deletes. Any CDC job into HBase ends with 
the rows that were deleted at the source still present.
   2. **Rows acknowledged by a checkpoint can be lost.** `prepareCommit()` 
returns `Optional.empty()` without flushing (`HbaseSinkWriter.java:110-112`). 
`HbaseClient` writes through a `BufferedMutator` sized by `write_buffer_size` 
(8 MB by default) and flushes it only in `close()` 
(`HbaseClient.java:330-334`). A checkpoint therefore completes while up to 8 MB 
of rows sit in the client buffer; if the task fails afterwards, those rows are 
gone and the restored job does not replay them.
   
   Neither case raises an error.
   
   ### SeaTunnel Version
   
   dev (`9cd7defe84`), 3.1.0
   
   ### SeaTunnel Config
   
   ```hocon
   env { job.mode = "STREAMING", checkpoint.interval = 10000 }
   source {
     MySQL-CDC {
       base-url = "jdbc:mysql://mysql:3306/app"
       table-names = ["app.orders"]
     }
   }
   sink {
     Hbase {
       zookeeper_quorum = "zk:2181"
       table = "orders"
       rowkey_column = ["id"]
       family_name { all_columns = info }
     }
   }
   ```
   
   A batch job with FakeSource rows of kind `INSERT`, `UPDATE_BEFORE`, 
`UPDATE_AFTER` and `DELETE` shows the first defect without a database: after 
the job the HBase table still holds the deleted row.
   
   ### Running Command
   
   ```shell
   ./bin/seatunnel.sh -c config/orders.conf
   ```
   
   ### Error Exception
   
   No exception. Deleted source rows remain in HBase; after a task failure the 
rows of the last checkpoint interval can be missing.
   
   ### Zeta or Flink or Spark Version
   
   Zeta
   
   ### Java or Scala Version
   
   11
   
   ### Screenshots
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://www.apache.org/foundation/policies/conduct).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to