Hi everyone,

Iceberg’s Load Table v2 proposal (
https://github.com/apache/iceberg/pull/17642) is moving forward. I’d like
to discuss what it means for Polaris and a few opportunities we should
consider.

   1.

   Supporting Iceberg v4

   The proposal adds a v2 load endpoint that supports v4 tables. The v1
   endpoint would reject v4 tables to protect older clients. For Polaris, this
   means adding the new endpoint and supporting metadata changes such as root
   manifests and relative paths.
   2.

   Storing table metadata in the metastore

   The proposal allows tables without a metadata JSON file location. This
   could let Polaris store table metadata directly in the metastore, avoiding
   the need to read and write a separate JSON file during loads and commits.
   This would be a big perf boost for every table commit and read. I’d love to
   see this happen.

   We would need a separate design for this, covering commit validation,
   metadata versioning, and how to export metadata when a table leaves
   Polaris. Then engine side changes are need as well, but it is more of
   Iceberg community’s concern.
   3.

   Defaulting snapshots to refs instead of all

   Large table metadata has been a recurring problem for users of Polaris
   and other catalogs, including Glue. This has also come up in several
   Iceberg community discussions.

   I think it a good time to propose making snapshots=refs the default.
   Returning only snapshots referenced by branches or tags would reduce
   response sizes for tables with long snapshot histories. Clients could still
   request all when they need the full set. This would help with response
   size, though reducing the cost of loading metadata inside Polaris would
   require additional work.

I suggest we start with v2 compatibility, explore metastore storage
separately, and raise the snapshot default upstream while the API is still
being discussed.

What do people think? Are there compatibility concerns we should consider?
Is there any other features in your mind for loadTable v2?

Thanks,
Yufei

Reply via email to