[
https://issues.apache.org/jira/browse/IMPALA-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107160#comment-18107160
]
Aleksandr Efimov commented on IMPALA-15253:
-------------------------------------------
[~stigahuang], I checked this against master before coming back to you.
IcebergScanNode.getNumInputRows() sums icebergMetadata().recordCount() over
fileDescs_, and IcebergScanNode does not override appendScanInputStats(), so
TScanInputStats.input_rows is never -1 for an Iceberg scan and the
missing-input_rows branch in the similarity check is unreachable there. That
override came in with IMPALA-14597 itself.
So the Iceberg side looks covered. What does still return -1 is the base
HdfsScanNode.getNumInputRows(), as soon as any selected partition has no
numRows - and there is a TODO on that very line about using HBO stats in that
case. Is that the gap you meant, or is there an Iceberg path I am not seeing?
Happy to take whichever it turns out to be.
> Get numRows from Iceberg stats
> ------------------------------
>
> Key: IMPALA-15253
> URL: https://issues.apache.org/jira/browse/IMPALA-15253
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Quanlong Huang
> Assignee: Aleksandr Efimov
> Priority: Major
>
> Currently, HBO uses numRows from HMS stats which could be missing. For
> Iceberg tables, stats always exist either in table level or file level. We
> should be able to get numRows for selected partitions (files) from Iceberg
> stats.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
