deniskuzZ commented on PR #5215: URL: https://github.com/apache/hive/pull/5215#issuecomment-2194719003
> > @zhangbutao, do you know if iceberg provides partition row_count stats? https://docs.google.com/document/d/1vaufuD47kMijz97LxM67X8OX-W2Wq7nmlz3jRo8J5Qk if not, maybe we can get it from meta table: > > ``` > > SELECT record_count FROM prod.db.table.partitions where spec_id in (....) > > ``` > > @deniskuzZ After some chek, i found that computing partitions stats by existing partitions metadata api which is used by `SELECT record_count FROM prod.db.table.partitions where spec_id in (....)` is very expensive if table has many data files, like the doc said: > >  > > The best way is that we can try to use some new partitons stats api added in Iceberg 1.5.0. We can do the partitions `row count` optimization after upgrading iceberg to 1.5.x. WDYT? i don't know the details, need to check the doc, but. if iceberg exposes record_count via partitions metatable why select would be expensive it's just 1 row fetch with spec filter? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For additional commands, e-mail: gitbox-h...@hive.apache.org
