yihua opened a new issue, #20464: URL: https://github.com/apache/hudi/issues/20464
Planning a snapshot read of a Hudi table today takes a chain of dependent object storage calls whose length does not depend on table size: timeline listing, replace and pending-plan metadata reads, then opening the metadata table (its own properties, timeline, `files` slice, HFile and log files), and a second pass for column statistics keyed by file name. A table with twenty files pays the same dozen-plus round trips as a table with a million, and every non-JVM engine has to reimplement the HFile reader, log block reader, and key hashing to consume it. RFC-117 proposes an optional, immutable snapshot manifest under `.hoodie/snapshot/`: one Parquet file holding every live file slice and its statistics for one snapshot, validated against the timeline listing the reader issues anyway, with fallback to the existing path when absent or stale. Target: two dependent round trips for planning on small and medium tables, on table versions 6 through 9, with no table config and no change to the commit protocol or the metadata table. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
