[
https://issues.apache.org/jira/browse/KUDU-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124204#comment-18124204
]
ASF subversion and git services commented on KUDU-3800:
-------------------------------------------------------
Commit fc1035f7a725a5cc915d7a4c764fee1c03f2cb8e in kudu's branch
refs/heads/master from Abhishek Chennaka
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=fc1035f7a ]
[Java] KUDU-3800: Add diff-scan option to return unobservable rows
The Java client side follow up work mirroring the C++ change. Adds
AsyncKuduScanner.DiffScanRowVisibility (OBSERVABLE_ONLY,
INCLUDE_UNOBSERVABLE) and a three argument diffScan() that takes it.
Unlike the C++ side the prior two argument diffScan() is kept, now
delegating with OBSERVABLE_ONLY. It cannot be dropped here: the
released flink-connector-kudu jar that kudu-replication binds to calls
it from KuduSplitGenerator, and that bytecode is never recompiled
against this tree, so removing the overload breaks continuous
replication at runtime with NoSuchMethodError.
When INCLUDE_UNOBSERVABLE is set the scanner sends the visibility on
NewScanRequestPB and advertises the DIFF_SCAN_ROW_VISIBILITY required
feature, so a server predating the feature rejects the scan rather
than silently dropping rows. Scan tokens record the dependency via
ScanTokenPB.Feature.RowVisibility, though feature flags in scan tokens
aren't verified as expected: see KUDU-3810, to be addressed in a
separate changelist.
KuduScanTokenBuilder.build() serializes straight from the builder
fields without running the AsyncKuduScanner constructor, so added the
checks done in the constructor here as well.
OBSERVABLE_ONLY is the wire default, so an OBSERVABLE_ONLY diff scan
emits the same requests and tokens as before this change.
Adds tests for the scanner and scan-token paths. UPSERT lifecycles
inside the diff window, wire-format assertions on the serialized
token, and a required-feature test pinning that OBSERVABLE_ONLY
advertises nothing new. The argument validation tests run against both
newScannerBuilder() and newScanTokenBuilder().
Change-Id: I9d61b2e942c9b2de1b46007f40311d1fb373104a
Reviewed-on: http://gerrit.cloudera.org:8080/24725
Reviewed-by: Alexey Serbin <[email protected]>
Tested-by: Alexey Serbin <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
> Diff scan drops rows whose full lifecycle is contained in the scan window
> -------------------------------------------------------------------------
>
> Key: KUDU-3800
> URL: https://issues.apache.org/jira/browse/KUDU-3800
> Project: Kudu
> Issue Type: Improvement
> Reporter: Abhishek Chennaka
> Assignee: Abhishek Chennaka
> Priority: Major
>
> Kudu's diff scan does not surface a row that is deleted at both endpoints of
> (snap_start, snap_end], even when it had a full INSERT → DELETE lifecycle
> inside the window. include_deleted_rows doesn't help — it only surfaces rows
> live at snap_start.
> This breaks two-tier consistency: a row previously migrated to a second tier
> and re-inserted-and-deleted inside a diff-scan window is never reported, so
> any stale copy downstream is never reconciled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
