JoeryH opened a new issue, #17257: URL: https://github.com/apache/iceberg/issues/17257
### Feature Request / Improvement By default Iceberg infers metrics for up to 100 columns (`write.metadata.metrics.max-inferred-column-defaults`), each with `truncate(16)` bounds. On wide tables this generates a large amount of per-data-file column stats, bloating manifests and slowing query planning. Users can tighten this by lowering the default or setting `write.metadata.metrics.column.*` to `none`/`counts` — but changing the config does **not** shrink existing manifests. `RewriteManifests` currently copies each data file's stats verbatim, so the now-excess stats persist until every affected data file is itself rewritten (expensive, and often unnecessary just to reclaim metadata space). Proposal: add a `pruneColumnStats()` option to the `RewriteManifests` API that drops column stats exceeding the table's current metrics config as manifests are rewritten: - columns configured `none` → all stats dropped - columns configured `counts` → bounds dropped - columns configured `truncate(n)` / `full` → unchanged (existing bounds are never re-truncated) Pruning applies only to files in manifests that are actually rewritten. When no clustering function is set via `clusterBy`, enabling it triggers a full rewrite so all data files are regrouped. This lets users reclaim manifest space and speed up planning after reducing metrics collection, without rewriting the data files themselves. ### Query engine None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
