morningman opened a new issue, #715: URL: https://github.com/apache/doris-flink-connector/issues/715
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris-flink-connector/issues?q=is%3Aissue) and found no similar issues. ### Version master @ 9550f9a0ebb16d95964ec65340f877b120f15403 (2026-11-05), with Doris master @ 72e91896bbdd8760f8bd4e488e5672406b40ce52. ### What's Wrong? With `source.scan.mode = 'initial'`, the snapshot phase is not pinned to the timestamp at which the incremental phase starts: - `DorisSourceSplitAssigner.create()` first plans the snapshot splits ([DorisSourceSplitAssigner.java#L58-L74](https://github.com/apache/doris-flink-connector/blob/9550f9a0ebb16d95964ec65340f877b120f15403/flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/source/assigners/DorisSourceSplitAssigner.java#L58-L74)), then takes the current Doris timestamp S as the incremental start ([#L150-L163](https://github.com/apache/doris-flink-connector/blob/9550f9a0ebb16d95964ec65340f877b120f15403/flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/source/assigners/DorisSourceSplitAssigner.java#L150-L163)). - Snapshot splits are read later with a plain `SELECT ... FROM db.tbl TABLET(...)` that has no TSO or version bound ([DorisFlightValueReader.java#L182-L212](https://github.com/apache/doris-flink-connector/blob/9550f9a0ebb16d95964ec65340f877b120f15403/flink-doris-connector/flink-doris-connector-base/src/main/java/org/apache/doris/flink/source/reader/DorisFlightValueReader.java#L182-L212)). Each split reads whatever is visible when it runs. - The first incremental window is `[S, now)`. This has three consequences: 1. **Overlap (always):** changes committed between S and the moment each snapshot split runs appear in the snapshot and again as change events. This happens without any failure. On Duplicate Key sources, or with append-only sinks, it produces real duplicate rows. The incremental-read documentation only describes replays after failures. 2. **Possible gap A:** a partition created after the snapshot is planned but before S is taken is not in any snapshot split, and its rows with a commit TSO below S are not in the incremental range either. 3. **Possible gap B:** a transaction whose commit TSO is below S, but which is not yet visible when its snapshot split runs, is missed by both phases. Incremental reads wait behind a visibility fence; snapshot reads do not. ### What You Expected? The snapshot and the incremental phase meet at the same point, with no gaps and no overlap. For example, read the snapshot at a version consistent with S, or take S only after the snapshot has been read under a visibility fence. Gaps A and B must not lose data. If the overlap is kept by design, the documentation should say so and point out that Duplicate Key sources need downstream deduplication. ### How to Reproduce? Overlap: 1. Create a Duplicate Key table with `changelog.enable = true` and enough data that the snapshot phase takes a while. 2. Start a job with `source.scan.mode = 'initial'`. 3. Insert new rows while snapshot splits are still running. 4. The inserted rows reach the sink twice. Gaps A and B are inferred from the code and still need confirmation. ### Anything Else? Found by code reading while documenting the incremental source; not yet reproduced. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
