Hi all,
Iceberg table has expire snapshots notion, which helps to delete snapshots that
are no longer needed along with data files, manifest and manifest lists:
// clean up the expired snapshots:
// 1. Get a list of the snapshots that were removed
// 2. Delete any data files that were deleted by those snapshots and
are not in the table
// 3. Delete any manifests that are no longer used by current snapshots
// 4. Delete the manifest lists
But we also have table metadata which is stored in JSON. New metadata version
is created for each metadata change.
I was assuming that with snapshot expiration operation, unneeded metadata files
will also be deleted but they are not.
My concern is that having JSON file for each metadata change with time may
consume lots of space (setting `iceberg.compress.metadata` to true can help but
not for long).
Is there an option to expire table metadata versions as well?
Kind regards,
Arina
