[
https://issues.apache.org/jira/browse/HIVE-29781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-29781 started by Denys Kuzmenko.
---------------------------------------------
> Iceberg: Basic stats fetch optimization - bulk partition stats read with
> query-scoped cache
> -------------------------------------------------------------------------------------------
>
> Key: HIVE-29781
> URL: https://issues.apache.org/jira/browse/HIVE-29781
> Project: Hive
> Issue Type: Bug
> Components: Iceberg integration
> Affects Versions: 4.2.0
> Reporter: Denys Kuzmenko
> Assignee: Denys Kuzmenko
> Priority: Major
> Labels: pull-request-available
>
> 1. Per-partition stats fetch is O(file) per partition.
> With hive.iceberg.stats.source=iceberg, StatsUtils.collectStatistics builds
> BasicStats per pruned partition. Each one called getPartishSummary, which
> opens the entire partition-stats file from object storage and linearly scans
> it for that single partition. N partitions ⇒ N full file opens + O(N²/2)
> record parses per TableScan.
>
>
> 2. The stats worker pool can't use the table-metadata cache (HIVE-24313
> interaction). The per-partition work runs on the dedicated basic-stats-*
> ForkJoin pool, whose thread factory doesn't propagate the SessionState
> ThreadLocal. Without it, SessionStateUtil.getQueryState(conf) is empty, so
> the handler's getTable() takes skipCache = orElse(true) — a full
> Catalogs.loadTable per partition (HMS RPC + metadata.json read from storage).
> That's each "Refreshing table metadata" line. The session-level cache is both
> unreadable and unpopulatable from those threads, so it never warms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
