[
https://issues.apache.org/jira/browse/HBASE-30351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junegunn Choi reassigned HBASE-30351:
-------------------------------------
Assignee: jeongmin kim
> ROWS_SCANNED scan metric double-counts rows passing the filter when the scan
> uses the joined heap (essential column family filtering)
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-30351
> URL: https://issues.apache.org/jira/browse/HBASE-30351
> Project: HBase
> Issue Type: Bug
> Components: metrics, Scanners
> Affects Versions: 3.0.0, 4.0.0-alpha-1, 2.6.7
> Reporter: Jeongmin Kim
> Assignee: jeongmin kim
> Priority: Minor
> Labels: pull-request-available
>
> When a scan enables {{Scan#setLoadColumnFamiliesOnDemand(true)}} and its
> filter marks some column families as non-essential
> ({{{}Filter#isFamilyEssential{}}}, introduced by HBASE-5416),
> {{RegionScannerImpl}} populates a row in two steps: the essential families
> through the store heap and — only if the row passes the filter — the
> remaining families through the joined heap.
> Both steps go through {{{}RegionScannerImpl#populateResult{}}}, which
> increments the ROWS_SCANNED metric
> ({{{}ServerSideScanMetrics.COUNT_OF_ROWS_SCANNED_KEY_METRIC_NAME{}}},
> introduced by HBASE-5980) each time the given heap finishes the current row:
> {code:java}
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> if (!moreCellsInRow) {
> incrementCountOfRowsScannedMetric(scannerContext);
> } {code}
> There is no guard against the second call on the joined heap, so every row
> that passes the filter and has data in a non-essential family is counted
> twice — once when the store heap finishes the row and once more when the
> joined heap finishes it. Rows rejected by the filter are counted once. The
> reported value effectively becomes "rows scanned + rows returned", which
> breaks the metric for its usual purposes (filter selectivity / scan
> efficiency analysis, the MapReduce {{ROWS_SCANNED}} HBase counter,
> comparisons against {{{}ROWS_FILTERED{}}}).
> Reproduction with stock filters only:
> # Create a table with two column families, e.g. {{essential}} and
> {{{}joined{}}}, and put a cell in both families for every row.
> # Scan with {{{}setScanMetricsEnabled(true){}}},
> {{setLoadColumnFamiliesOnDemand(true)}} and a {{SingleColumnValueFilter}} on
> family {{essential}} with {{setFilterIfMissing(true)}} (this makes the other
> family non-essential via {{{}SingleColumnValueFilter#isFamilyEssential{}}}).
> # With 10 rows of which 5 match the filter, ROWS_SCANNED reports 15 instead
> of 10, while ROWS_FILTERED correctly reports 5. Without
> {{setLoadColumnFamiliesOnDemand(true)}} the same scan reports 10.
> The double counting is invisible in the existing
> {{TestServerSideScanMetricsFromClientSide}} because its table has a single
> column family, so the joined heap is never exercised.
> Proposed fix: count a completed row in {{populateResult}} only when
> populating from the store heap. The joined heap only re-populates rows that
> already passed the filter, so the store heap completion is the single
> canonical "row scanned" event (rows filtered by row key keep their separate
> increment in {{{}nextInternal{}}}). A PR with the fix and a unit test in
> {{TestServerSideScanMetricsFromClientSide}} follows.
> Verified on branch-2, branch-2.6, branch-3 and master (same code in
> {{populateResult}} / {{{}populateFromJoinedHeap{}}}).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
