amogh-jahagirdar opened a new pull request, #17347: URL: https://github.com/apache/iceberg/pull/17347
Add a Spark write configuration that makes streaming appends use the regular merge append instead of fast append when enabled. Defaults to false, preserving existing fast-append behavior. The rationale here is that as of today fast appends produces many tiny manifests, and there are Spark Streaming commit intervals where the latency of merging manifests is a negligible cost compared to the maintenance and read burden (and metadata bloat/footprint). This is a metadata equivalent to a lot of the read/write tradeoff analysis we did for DVs and equality deletes, where the notion of a "cheap write" ends up being compromised when there isn't very aggressive maintenance (which oftentimes there isn't). Furthermore streaming engine integrations like Flink and Kafka Connect today use merge appends so I think there's general precedent that this is probably good enough for the streaming workloads Iceberg is capable of handling today, especially if it's opt in. Especially since going forward in V4, we're addressing these cases in a much better way at the table format level I do think at some point we should flip this default (a future release version), but in the interim to prevent any unforseen regressions on upgrade, I think it makes sense to default to false. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
