Hello Marton Greber, Alexey Serbin, Zoltan Martonka, Ashwani Raina, Kudu
Jenkins, Michael Smith,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24578
to look at the new patch set (#16).
Change subject: KUDU-3800 Add diff-scan option to return unobservable rows
......................................................................
KUDU-3800 Add diff-scan option to return unobservable rows
A diff scan reports rows whose state changed inside the
(snap_start_timestamp, snap_timestamp] range. Rows live at
snap_start_timestamp and deleted before snap_timestamp are surfaced
via 'include_deleted_rows'; rows inserted or updated inside the range
that are still live at snap_timestamp are surfaced by default. But rows
whose entire lifecycle (INSERT -> eventual DELETE) is contained inside
the range exist at neither endpoint, so no snapshot read would see
them, and 'include_deleted_rows' does not apply. It only surfaces
rows that were live at snap_start_timestamp. The diff scan drops them
even though a two-tier system that uses diff scans to keep a second
tier consistent with Kudu would want them: a row previously migrated
out of Kudu that is re-inserted and re-deleted inside a diff-scan
window is not reported today, so any stale copy downstream is never
reconciled.
Note: This is NOT change-data-capture. The diff scan collapses all
mutations for a row within the range into a single output row
reflecting its state at snap_timestamp; intermediate UPDATE values and
mutation counts are not preserved. Callers needing per-mutation
history need a different mechanism.
This patch adds an optional 'row_visibility' knob to the
NewScanRequestPB and plumbs it through:
- ScanConfiguration / KuduScanner. A new SetDiffScan overload takes a
'DiffScanRowVisibility' enum (OBSERVABLE_ONLY / INCLUDE_UNOBSERVABLE)
so the intent is legible at call sites.
- ScanTokenPB and KuduScanTokenBuilder, so token-based clients can carry
the flag over the wire.
- RowIteratorOptions -> MemRowSet iterator and DeltaApplier /
SelectedDeltas, where the "drop unobservable" test now short-circuits
when the caller opts in. Such rows are returned marked deleted via the
IS_DELETED virtual column.
The tablet server enforces the invariant uniformly: it returns
INVALID_SCAN_SPEC if the flag is set without snap_start_timestamp
(regardless of read mode), and on diff scans whose projection lacks
an IS_DELETED virtual column. DCHECKs in DeltaApplier and MemRowSet
catch in-process callers that build a RowIteratorOptions with the flag
set outside a valid diff-scan configuration.
The RPC field is a new 'optional enum' and is wire-compatible with
older servers (which ignore it) and older clients (which don't set it).
Corresponding tests have been added in tablet_server-test.cc, client-test.cc,
scan_token-test.cc and diff_scan-test.cc.
The prior two argument SetDiffScan() (which was never part of the client ABI)
is now replaced with three argument SetDiffScan() and exported for user code
but documented private.
Followup work to include Java client support.
Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
---
M src/kudu/client/client-test.cc
M src/kudu/client/client.cc
M src/kudu/client/client.h
M src/kudu/client/client.proto
M src/kudu/client/scan_configuration.cc
M src/kudu/client/scan_configuration.h
M src/kudu/client/scan_token-internal.cc
M src/kudu/client/scan_token-test.cc
M src/kudu/client/scanner-internal.cc
M src/kudu/common/common.proto
M src/kudu/integration-tests/fuzz-itest.cc
M src/kudu/tablet/delta_applier.cc
M src/kudu/tablet/delta_store.cc
M src/kudu/tablet/delta_store.h
M src/kudu/tablet/diff_scan-test.cc
M src/kudu/tablet/memrowset.cc
M src/kudu/tablet/rowset.cc
M src/kudu/tablet/rowset.h
M src/kudu/tablet/tablet.h
M src/kudu/tserver/tablet_server-test.cc
M src/kudu/tserver/tablet_service.cc
M src/kudu/tserver/tserver.proto
22 files changed, 925 insertions(+), 31 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/24578/16
--
To view, visit http://gerrit.cloudera.org:8080/24578
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0d0727fb2b89c727c669ec218e3bb8594336359a
Gerrit-Change-Number: 24578
Gerrit-PatchSet: 16
Gerrit-Owner: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Abhishek Chennaka <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
