Abhishek Chennaka has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/24578 )
Change subject: Add diff-scan option to return unobservable rows ...................................................................... Add diff-scan option to return unobservable rows A diff scan reports rows whose state changed inside the (snap_start_timestamp, snap_timestamp] range. Rows live at snap_start_timestamp and deleted before snap_timestamp are surfaced via 'include_deleted_rows'; rows inserted or updated inside the range that are still live at snap_timestamp are surfaced by default. But rows whose entire lifecycle (INSERT -> eventual DELETE) is contained inside the range exist at neither endpoint, so no snapshot read would see them, and 'include_deleted_rows' does not apply. It only surfaces rows that were live at snap_start_timestamp. The diff scan drops them even though a change-data-capture pipeline reconstructing row-level history would want them. This patch adds an optional 'include_unobservable_rows' knob to the NewScanRequestPB and plumb it through: - ScanConfiguration / KuduScanner. A new SetDiffScan overload takes a 'DiffScanRowVisibility' enum (OBSERVABLE_ONLY / INCLUDE_UNOBSERVABLE) rather than a raw bool, so the intent is legible at call sites. - ScanTokenPB and KuduScanTokenBuilder, so token-based clients can carry the flag over the wire. - RowIteratorOptions -> MemRowSet iterator and DeltaApplier / SelectedDeltas, where the "drop unobservable" test now short-circuits when the caller opts in. Such rows are returned marked deleted via the IS_DELETED virtual column. The tablet server enforces the invariant uniformly: it returns INVALID_SCAN_SPEC if the flag is set without snap_start_timestamp (regardless of read mode), and on diff scans whose projection lacks an IS_DELETED virtual column. DCHECKs in DeltaApplier and MemRowSet catch in-process callers that build a RowIteratorOptions with the flag set outside a valid diff-scan configuration. The RPC field is a new 'optional bool' and is wire-compatible with older servers (which ignore it) and older clients (which don't set it). Corresponding tests have been added in client-test.cc and diff_scan-test.cc. Followup work to include Java and Python client support. Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a --- M src/kudu/client/client-test.cc M src/kudu/client/client.cc M src/kudu/client/client.h M src/kudu/client/client.proto M src/kudu/client/scan_configuration.cc M src/kudu/client/scan_configuration.h M src/kudu/client/scan_token-internal.cc M src/kudu/client/scanner-internal.cc M src/kudu/tablet/delta_applier.cc M src/kudu/tablet/delta_store.cc M src/kudu/tablet/delta_store.h M src/kudu/tablet/diff_scan-test.cc M src/kudu/tablet/memrowset.cc M src/kudu/tablet/rowset.cc M src/kudu/tablet/rowset.h M src/kudu/tablet/tablet.h M src/kudu/tserver/tablet_service.cc M src/kudu/tserver/tserver.proto 18 files changed, 564 insertions(+), 19 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/24578/7 -- To view, visit http://gerrit.cloudera.org:8080/24578 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a Gerrit-Change-Number: 24578 Gerrit-PatchSet: 7 Gerrit-Owner: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Ashwani Raina <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Marton Greber <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
