alexprosak opened a new pull request, #18068: URL: https://github.com/apache/iceberg/pull/18068
Closes https://github.com/apache/iceberg/issues/13188 Implements an initial-snapshot bootstrap for streaming reads of Iceberg tables in Spark. A streaming query started with a fresh checkpoint now reads the current snapshot in full before continuing incrementally - matching how Delta Lake's streaming source behaves by default. Currently a fresh Iceberg streaming query without options replays the table's history from the oldest ancestor snapshot, which is slow for long-lived tables and any expired/deleted snapshots cannot be streamed. The only existing workarounds (bootstrap via batch read + cut over by timestamp) are fragile, so users can't easily express the common pattern of loading the entire table and then keeping up with new append data. ## About the Change This PR adds a `stream-from-snapshot` option: | `stream-from-snapshot` | Behavior | |---|---| | *(not set)* | **new default**: read the current snapshot in full, then continue with new snapshots | | `latest` | read snapshots committed after latest on stream startup | | `earliest` | read from the oldest ancestor (the existing default) | | *`<snapshot-id>`* | read snapshots after the given snapshot id (exclusive) | `stream-from-snapshot` and `stream-from-timestamp` are mutually exclusive. Previous default behavior to read from oldest ancestor can be opted-in via `stream-from-snapshot=earliest` **Follow-ups**: - Docs update with new default behavior & `stream-from-snapshot` option - Ports to other supported spark versions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
