Hi devs, I'm experimenting with Apache Iceberg for Structured Streaming sink - plan to experiment with source as well, but I see PR still in review.
It seems that "fast append" pretty much helps to retain reasonable latency for committing, though the metadata directory grows too fast. I found the option 'write.metadata.delete-after-commit.enabled' (false by default), and disabled it, and the overall size looks fine afterwards. That said, given the option is false by default, I'm wondering which would be impacted when turning off this option. My understanding is that it doesn't affect time-travel (as it refers to a snapshot), and restoring is also from snapshot, so not sure which point to consider when turning on the option. Thanks, Jungtaek Lim
