Andrew Kyle Purtell created PHOENIX-8044:
--------------------------------------------
Summary: Phoenix skip scan queries can fail with
IllegalStateException
Key: PHOENIX-8044
URL: https://issues.apache.org/jira/browse/PHOENIX-8044
Project: Phoenix
Issue Type: Bug
Affects Versions: 5.3.2
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
Fix For: 5.4.0, 5.3.3
During read repair of unverified rows on covered global indexes, Phoenix skip
scan queries can fail with {{IllegalStateException}} indicating that a cell
seek hint does not strictly advance ({{{}"The next hint must come after
previous hint"{}}}). This failure is due to state pollution when the primary
scan and a nested read repair scan concurrently mutate a shared chain of
{{Filter}} structures.
The failure happens under a set of specific conditions:
- *Covered Global Index Traversal:* The query targets a covered global index
where {{GlobalIndexChecker}} wraps the underlying region scanner.
- *Skip-Scan Optimization:* The query execution plan employs a
{{SkipScanFilter}} (such as evaluating an {{IN}} predicate over leading indexed
columns).
- *Unverified Point Mutations:* An intervening index row physically sorted
between two active skip scan targets remains unverified because of an earlier
failed secondary write phase. Point rebuilding fails to verify its latest cell
version.
- *Immediate Boundary Proximity:* The unverified row is the first entry
evaluated by the main scanner immediately following a matched skip scan key.
- *Column Encoded Row Layout:* The empty column cell sorts first (guaranteed
on tables with column encoding). {{UnverifiedRowFilter}} evaluates this cell,
identifies the row as unverified, and immediately returns
{{{}INCLUDE_AND_SEEK_NEXT_ROW{}}}. Consequently, the downstream
{{SkipScanFilter}} in the filter chain is bypassed on this initial pass.
- *Trailing Intermediary Rows:* At least one subsequent index row exists that
also sorts prior to the next skip scan seek target.
{{GlobalIndexScanner}} constructs its internal scanners, including those for
read repair, using HBase's copy constructor {{{}new Scan(scan){}}}. This
constructor only performs a shallow copy of the configured {{Filter}}
reference. As a result, the primary scanner and all internal downstream
scanners share the same {{Filter}} instances. While sharing a filter chain is
valid for scanners that sequentially execute scans, this is not safe for
concurrent use cases.
Here is an example of this going awry: When main scanner encounters the
unverified row, because the empty column appears first, {{UnverifiedRowFilter}}
intercepts the row and issues an immediate row-advance directive without
invoking client side filters, so the shared {{SkipScanFilter}} remains unaware
of this row. When point rebuilding leaves the row unverified,
{{repairIndexRows()}} launches {{singleRowIndexScan}} over the shared filter
chain to inspect previous cell versions. This nested scan delegates directly to
{{{}SkipScanFilter{}}}. The filter evaluates the key, computes a seek hint
targeting the next skip-scan interval, caches the generated hint, and
terminates the single row scan. The primary scanner resumes and reads the next
index row, which still precedes the target skip scan key. When the main scanner
calls {{{}filterCell(){}}}, the shared {{SkipScanFilter}} emits the same seek
hint cached during the repair scan. HBase's scanner infrastructure enforces
monotonic progression via {{{}setNextCellHint(){}}}, requiring every hint to
sort strictly after the prior hint ({{{}next > prev{}}}). Because {{{}next ==
prev{}}}, the scanner aborts with an unhandled exception.
The fix is to avoid sharing {{Filter}} instances. This is a significant general
safety improvement because these structures are not designed to be shared.
During initialization of the repair path, {{GlobalIndexChecker}} serializes the
client filter hierarchy once via {{{}ProtobufUtil{}}}. Each subsequent
{{singleRowIndexScan}} deserializes a fresh filter instance from this byte
payload and wraps it with dedicated {{UnverifiedRowFilter}} and
{{PagingFilter}} instances. This provides complete state isolation for skip
scan cursors across point lookups at the cost of a single protobuf parse per
repair, which is negligible given the rarity of read repairs.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
