Jeongmin Kim created HBASE-30351:
------------------------------------
Summary: ROWS_SCANNED scan metric double-counts rows passing the
filter when the scan uses the joined heap (essential column family filtering)
Key: HBASE-30351
URL: https://issues.apache.org/jira/browse/HBASE-30351
Project: HBase
Issue Type: Bug
Components: metrics, Scanners
Affects Versions: 2.6.7, 3.0.0, 4.0.0-alpha-1
Reporter: Jeongmin Kim
When a scan enables {{Scan#setLoadColumnFamiliesOnDemand(true)}} and its filter
marks some column families as non-essential ({{{}Filter#isFamilyEssential{}}},
introduced by HBASE-5416), {{RegionScannerImpl}} populates a row in two steps:
the essential families through the store heap and — only if the row passes the
filter — the remaining families through the joined heap.
Both steps go through {{{}RegionScannerImpl#populateResult{}}}, which
increments the ROWS_SCANNED metric
({{{}ServerSideScanMetrics.COUNT_OF_ROWS_SCANNED_KEY_METRIC_NAME{}}},
introduced by HBASE-5980) each time the given heap finishes the current row:
{code:java}
nextKv = heap.peek();
moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
if (!moreCellsInRow) {
incrementCountOfRowsScannedMetric(scannerContext);
} {code}
There is no guard against the second call on the joined heap, so every row that
passes the filter and has data in a non-essential family is counted twice —
once when the store heap finishes the row and once more when the joined heap
finishes it. Rows rejected by the filter are counted once. The reported value
effectively becomes "rows scanned + rows returned", which breaks the metric for
its usual purposes (filter selectivity / scan efficiency analysis, the
MapReduce {{ROWS_SCANNED}} HBase counter, comparisons against
{{{}ROWS_FILTERED{}}}).
Reproduction with stock filters only:
# Create a table with two column families, e.g. {{essential}} and
{{{}joined{}}}, and put a cell in both families for every row.
# Scan with {{{}setScanMetricsEnabled(true){}}},
{{setLoadColumnFamiliesOnDemand(true)}} and a {{SingleColumnValueFilter}} on
family {{essential}} with {{setFilterIfMissing(true)}} (this makes the other
family non-essential via {{{}SingleColumnValueFilter#isFamilyEssential{}}}).
# With 10 rows of which 5 match the filter, ROWS_SCANNED reports 15 instead of
10, while ROWS_FILTERED correctly reports 5. Without
{{setLoadColumnFamiliesOnDemand(true)}} the same scan reports 10.
The double counting is invisible in the existing
{{TestServerSideScanMetricsFromClientSide}} because its table has a single
column family, so the joined heap is never exercised.
Proposed fix: count a completed row in {{populateResult}} only when populating
from the store heap. The joined heap only re-populates rows that already passed
the filter, so the store heap completion is the single canonical "row scanned"
event (rows filtered by row key keep their separate increment in
{{{}nextInternal{}}}). A PR with the fix and a unit test in
{{TestServerSideScanMetricsFromClientSide}} follows.
Verified on branch-2, branch-2.6, branch-3 and master (same code in
{{populateResult}} / {{{}populateFromJoinedHeap{}}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
