goutamadwant opened a new issue, #13244:
URL: https://github.com/apache/seatunnel/issues/13244
### Search before asking
- [x] I had searched in the issues and found no similar issues.
### What happened
Two defects in `HbaseSinkWriter`, both on dev (`9cd7defe84`) and in 3.1.0:
1. **Deleted rows come back.** `write()` turns every row into a `Put`
(`HbaseSinkWriter.java:104-107`); there is no `RowKind` check anywhere in the
connector. A CDC `DELETE` row still carries the old column values, so the sink
writes them back as a normal row. An `UPDATE_BEFORE` row is written the same
way and is then overwritten by the `UPDATE_AFTER` for the same key, which hides
the problem for updates but not for deletes. Any CDC job into HBase ends with
the rows that were deleted at the source still present.
2. **Rows acknowledged by a checkpoint can be lost.** `prepareCommit()`
returns `Optional.empty()` without flushing (`HbaseSinkWriter.java:110-112`).
`HbaseClient` writes through a `BufferedMutator` sized by `write_buffer_size`
(8 MB by default) and flushes it only in `close()`
(`HbaseClient.java:330-334`). A checkpoint therefore completes while up to 8 MB
of rows sit in the client buffer; if the task fails afterwards, those rows are
gone and the restored job does not replay them.
Neither case raises an error.
### SeaTunnel Version
dev (`9cd7defe84`), 3.1.0
### SeaTunnel Config
```hocon
env { job.mode = "STREAMING", checkpoint.interval = 10000 }
source {
MySQL-CDC {
base-url = "jdbc:mysql://mysql:3306/app"
table-names = ["app.orders"]
}
}
sink {
Hbase {
zookeeper_quorum = "zk:2181"
table = "orders"
rowkey_column = ["id"]
family_name { all_columns = info }
}
}
```
A batch job with FakeSource rows of kind `INSERT`, `UPDATE_BEFORE`,
`UPDATE_AFTER` and `DELETE` shows the first defect without a database: after
the job the HBase table still holds the deleted row.
### Running Command
```shell
./bin/seatunnel.sh -c config/orders.conf
```
### Error Exception
No exception. Deleted source rows remain in HBase; after a task failure the
rows of the last checkpoint interval can be missing.
### Zeta or Flink or Spark Version
Zeta
### Java or Scala Version
11
### Screenshots
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]