Hello Alexey Serbin, Ashwani Raina, Kudu Jenkins, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24578
to look at the new patch set (#5).
Change subject: Add diff-scan option to return unobservable rows
......................................................................
Add diff-scan option to return unobservable rows
A diff scan reports rows whose state changed inside the
(snap_start_timestamp, snap_timestamp] range. Rows live at
snap_start_timestamp and deleted before snap_timestamp are surfaced
via 'include_deleted_rows'; rows inserted or updated inside the range
that are still live at snap_timestamp are surfaced by default. But rows
whose entire lifecycle (INSERT -> eventual DELETE) is contained inside
the range exist at neither endpoint, so no snapshot read would see
them, and 'include_deleted_rows' does not apply. It only surfaces
rows that were live at snap_start_timestamp. The diff scan drops them
even though a change-data-capture pipeline reconstructing row-level
history would want them.
This patch adds an optional 'include_unobservable_rows' knob to the
NewScanRequestPB and plumb it through:
- ScanConfiguration / KuduScanner. A new SetDiffScan overload takes a
'DiffScanRowVisibility' enum (OBSERVABLE_ONLY / INCLUDE_UNOBSERVABLE)
rather than a raw bool, so the intent is legible at call sites.
- ScanTokenPB and KuduScanTokenBuilder, so token-based clients can carry
the flag over the wire.
- RowIteratorOptions -> MemRowSet iterator and DeltaApplier /
SelectedDeltas, where the "drop unobservable" test now short-circuits
when the caller opts in. Such rows are returned marked deleted via the
IS_DELETED virtual column.
The tablet server enforces the invariant uniformly: it returns
INVALID_SCAN_SPEC if the flag is set without snap_start_timestamp
(regardless of read mode), and on diff scans whose projection lacks
an IS_DELETED virtual column. DCHECKs in DeltaApplier and MemRowSet
catch in-process callers that build a RowIteratorOptions with the flag
set outside a valid diff-scan configuration.
The RPC field is a new 'optional bool' and is wire-compatible with
older servers (which ignore it) and older clients (which don't set it).
Corresponding tests have been added in client-test.cc and
diff_scan-test.cc.
Followup work to include Java and Python client support.
Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
---
M src/kudu/client/client-test.cc
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/client.proto
M src/kudu/client/scan_configuration.cc
M src/kudu/client/scan_configuration.h
M src/kudu/client/scan_token-internal.cc
M src/kudu/client/scanner-internal.cc
M src/kudu/tablet/delta_applier.cc
M src/kudu/tablet/delta_store.cc
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/memrowset.cc
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/tablet.h
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver.proto
18 files changed, 558 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/24578/5
--
To view, visit http://gerrit.cloudera.org:8080/24578
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
Gerrit-Change-Number: 24578
Gerrit-PatchSet: 5
Gerrit-Owner: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Michael Smith <[email protected]>
