litiliu commented on code in PR #4608:
URL: https://github.com/apache/flink-cdc/pull/4608#discussion_r4548734427
##########
flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-fluss/src/main/java/org/apache/flink/cdc/connectors/fluss/source/reader/FlussSourceReader.java:
##########
@@ -243,7 +263,7 @@ protected void onSplitFinished(Map<String, FlussSplitState>
finishedSplitIds) {
private void activateSplits(List<FlussSplitBase> splits) {
List<FlussSplitBase> activeSplits = new ArrayList<>();
for (FlussSplitBase split : splits) {
- if (isActive(split.getTablePath())) {
+ if (isSplitActive(split)) {
Review Comment:
@fxbing Thanks for pointing this out. Repeated “not found” results cannot
reliably distinguish deletion from delayed visibility.
Propagating PartitionNotExistException avoids prematurely finishing the
Reader’s split, but restarting is not a complete fix. During recovery, a stale
discovery result caused by delayed metadata visibility could make the
Enumerator request removal of a live partition. The Reader would then discard
its restored split. Later rediscovery may initialize a new offset instead of
resuming from the checkpoint.
listTables() currently also queries a randomly selected TabletServer.
Consecutive calls may reach different servers and observe different metadata
views.
A temporary omission could cause CDC to unsubscribe a table it is already
reading. If rediscovered after cleanup, the table may be reinitialized and
previously consumed data replayed, depending on the startup mode.
We therefore need an authoritative API to confirm table or partition
deletion, with a clear consistency guarantee. WDYT? @loserwang1024
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
