[
https://issues.apache.org/jira/browse/IMPALA-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18105814#comment-18105814
]
Aleksandr Efimov commented on IMPALA-15253:
-------------------------------------------
[~stigahuang], I started looking at this and want to check what you had in
mind, because part of it may already be in place.
IcebergScanNode overrides getNumInputRows() and sums
icebergMetadata().recordCount() over the selected file descriptors, so
TScanInputStats.input_rows is never -1 for an Iceberg scan and the run never
falls back to the exact-match path. That override arrived with IMPALA-14597
itself.
Is there a path I'm missing where an Iceberg table still ends up with HMS row
counts — a scan that isn't IcebergScanNode, or table-level numRows used
elsewhere — or did the initial patch already cover this?
Separately, while reading that code I noticed something adjacent, and I'd
rather ask than file noise: TABLESAMPLE doesn't reach HBO at all. The scan key
is table name + conjuncts + LIMIT, and IcebergScanNode.getNumInputRows() sums
all of fileDescs_ while computeCardinalities() right next to it uses
sampledFiles_. So a sampled run and a full run of the same query share a key
and report the same input rows, and a cardinality learned from the sample can
then be applied to the full scan. HdfsScanNode is partly covered through
sampledPartitions_, but file-level sampling inside a partition isn't reflected
there either. I haven't reproduced it on a cluster — if it looks real to you,
I'm happy to file it and take it.
> Get numRows from Iceberg stats
> ------------------------------
>
> Key: IMPALA-15253
> URL: https://issues.apache.org/jira/browse/IMPALA-15253
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Quanlong Huang
> Assignee: Aleksandr Efimov
> Priority: Major
>
> Currently, HBO uses numRows from HMS stats which could be missing. For
> Iceberg tables, stats always exist either in table level or file level. We
> should be able to get numRows for selected partitions (files) from Iceberg
> stats.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
