[
https://issues.apache.org/jira/browse/IMPALA-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-14879:
-------------------------------------
Description:
Currently Impala always reads the whole table during COMPUTE STATS (unless
table sample is used).
It is possible to provide efficient incremental implementation in Iceberg (as
long as deletes are ignored).
- COMPUTE STATS should save stats in Puffin (e.g. Theta sketches for NDV) for
the snapshot where the stats were computed
- subsequent COMPUTE STATS only need to read new files and merge the results
with the one in the snapshot
This would be both easier to use and more efficient than partition based
COMPUTE INCREMENTAL STATS for Hive tables.
One issue to solve is how to store statistics not handled by Iceberg, e.g.
max/avg length of strings.
The following design doc has detailed discussion of stats usage of different
engines in Iceberg, but did not come to a conclusion:
https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM/edit?tab=t.0#heading=h.hg9vbmr09jb8
I see 3 ways to solve this:
1. wait till Iceberg has a solution for all of Impala's stats
2. write non-Icerberg compatible stats in an Impala specific key in Puffin
3. keep storing missing stats in HMS, but improve them to provide incremental
support (e.g. store snapshot id)
was:Currently Impala always reads the whole table during COMPUTE STATS
(unless table sample is used). It is possible to provide efficient incremental
implementation in Iceberg (as long as deletes are ignored) by saving stats in
Puffin (e.g. Theta sketches for NDV) for the snapshot where the stats were
computed. Subsequent COMPUTE STATS calls only need to read new file and merge
the results with one in the snapshot.
> Support incremental stats in Iceberg tables with Puffin files
> -------------------------------------------------------------
>
> Key: IMPALA-14879
> URL: https://issues.apache.org/jira/browse/IMPALA-14879
> Project: IMPALA
> Issue Type: Epic
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: iceberg
>
> Currently Impala always reads the whole table during COMPUTE STATS (unless
> table sample is used).
> It is possible to provide efficient incremental implementation in Iceberg (as
> long as deletes are ignored).
> - COMPUTE STATS should save stats in Puffin (e.g. Theta sketches for NDV)
> for the snapshot where the stats were computed
> - subsequent COMPUTE STATS only need to read new files and merge the results
> with the one in the snapshot
> This would be both easier to use and more efficient than partition based
> COMPUTE INCREMENTAL STATS for Hive tables.
> One issue to solve is how to store statistics not handled by Iceberg, e.g.
> max/avg length of strings.
> The following design doc has detailed discussion of stats usage of different
> engines in Iceberg, but did not come to a conclusion:
> https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM/edit?tab=t.0#heading=h.hg9vbmr09jb8
> I see 3 ways to solve this:
> 1. wait till Iceberg has a solution for all of Impala's stats
> 2. write non-Icerberg compatible stats in an Impala specific key in Puffin
> 3. keep storing missing stats in HMS, but improve them to provide incremental
> support (e.g. store snapshot id)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
