[
https://issues.apache.org/jira/browse/PHOENIX-4277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16193400#comment-16193400
]
James Taylor edited comment on PHOENIX-4277 at 10/6/17 1:20 AM:
----------------------------------------------------------------
[~vincentpoon] - after further discussion, another easier option surfaced.
HBase has the ability to run a scan with KEEP_DELETED_CELLS enabled. Phoenix
could leverage this for any point-in-time scan (i.e. a scan with a max time
range set to a value other than LATEST_TIMESTAMP), so no need for any new scan
attribute or client-side change, since the intent with point-in-time scans in
Phoenix is to treat delete markers consistently with updates (i'm not sure of
the logic of seeing prior update version which would disappear with compaction
while *not* seeing deletes).
Here's the new coprocessor we could add to BaseScannerRegionObserver:
{code}
@Override
public KeyValueScanner preStoreScannerOpen(final
ObserverContext<RegionCoprocessorEnvironment> c,
final Store store, final Scan scan, final NavigableSet<byte[]> targetCols,
final KeyValueScanner s) throws IOException {
if (scan.isRaw() || scan.getTimeRange().getMax() ==
HConstants.LATEST_TIMESTAMP) {
return s;
}
ScanInfo oldScanInfo = store.getScanInfo();
ScanInfo scanInfo =
new ScanInfo(oldScanInfo.getFamily(),
Math.max(oldScanInfo.getMinVersions(), 1),
oldScanInfo.getMaxVersions(), oldScanInfo.getTtl(),
KeepDeletedCells.TRUE,
oldScanInfo.getTimeToPurgeDeletes(), oldScanInfo.getComparator());
return new StoreScanner(store, scanInfo, scan, targetCols,
c.getEnvironment().getRegion().getReadpoint(scan.getIsolationLevel()));
}
{code}
This will prevent us from having to replicate in Phoenix the filtering HBase
does for deletes in a user-level Filter (which the HBase community tried, but
failed to do already).
We'd need to investigate how to do this for snapshot reads, but the above
should fix the current issue with the IndexScrutinyTool.
was (Author: jamestaylor):
[~vincentpoon] - after further discussion, another easier option surfaced.
HBase has the ability to run a scan with KEEP_DELETED_CELLS enabled. Phoenix
could leverage this for any point-in-time scan (i.e. a scan with a max time
range set to a value other than LATEST_TIMESTAMP), so no need for any new scan
attribute or client-side change, since the intent with point-in-time scans in
Phoenix is to treat delete markers consistently with updates (i'm not sure of
the logic of seeing prior update version which would disappear with compaction
while *not* seeing deletes).
Here's the new coprocessor we could add to BaseScannerRegionObserver:
{code}
@Override
public KeyValueScanner preStoreScannerOpen(final
ObserverContext<RegionCoprocessorEnvironment> c,
final Store store, final Scan scan, final NavigableSet<byte[]> targetCols,
final KeyValueScanner s) throws IOException {
if (!isSOREnabled || !scan.isRaw() || scan.getTimeRange().getMax() ==
HConstants.LATEST_TIMESTAMP) {
return s;
}
ScanInfo oldScanInfo = store.getScanInfo();
ScanInfo scanInfo =
new ScanInfo(oldScanInfo.getFamily(),
Math.max(oldScanInfo.getMinVersions(), 1),
oldScanInfo.getMaxVersions(), oldScanInfo.getTtl(),
KeepDeletedCells.TRUE,
oldScanInfo.getTimeToPurgeDeletes(), oldScanInfo.getComparator());
return new StoreScanner(store, scanInfo, scan, targetCols,
c.getEnvironment().getRegion().getReadpoint(scan.getIsolationLevel()));
}
{code}
This will prevent us from having to replicate in Phoenix the filtering HBase
does for deletes in a user-level Filter (which the HBase community tried, but
failed to do already).
We'd need to investigate how to do this for snapshot reads, but the above
should fix the current issue with the IndexScrutinyTool.
> Treat delete markers consistently with puts for point-in-time scans
> -------------------------------------------------------------------
>
> Key: PHOENIX-4277
> URL: https://issues.apache.org/jira/browse/PHOENIX-4277
> Project: Phoenix
> Issue Type: Bug
> Reporter: James Taylor
> Assignee: Vincent Poon
>
> The IndexScrutinyTool relies on doing point-in-time scans to determine
> consistency between the index and data tables. Unfortunately, deletes to the
> tables cause a problem with this approach, since delete markers take effect
> even if they're at a later time stamp than the point-in-time at which the
> scan is being done (unless KEEP_DELETED_CELLS is true). The logic of this is
> that scans should get the same results before and after a compaction take
> place.
> Taking snapshots does not help with this since they cannot be taken at a
> point-in-time and the delete markers will act the same way - there's no way
> to guarantee that the index and data table snapshots have the same "logical"
> set of data.
> Using raw scans would allow us to see the delete markers and do the correct
> point-in-time filtering ourselves. We'd need to write the filters to do this
> correctly (see the Tephra TransactionVisibilityFilter for an implementation
> of this that could be adapted). We'd also need to hook this into Phoenix or
> potentially dip down to the HBase level to do this.
> Thanks for brainstorming on this with me, [~lhofhansl].
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
