brishi19791 opened a new pull request, #791: URL: https://github.com/apache/incubator-xtable/pull/791
This PR removes redundant DeltaLog.getSnapshotAt(version) calls in the Delta source conversion path that were happening for every commit. getSnapshotAt can internally trigger an expensive Spark job and associated network I/O (e.g., listing/reading Delta log metadata from remote storage) to resolve the snapshot for a given version. We now fetch the snapshot once per commit/version and reuse it to construct the InternalTable (via a DeltaTableExtractor.table(Snapshot, tableName) overload), instead of re-resolving the same snapshot multiple times. Impact - Avoids redundant snapshot resolution work per commit/version (and the Spark job + network calls it may trigger). - Reduces end-to-end conversion latency, especially for large commit backlogs. - No intended functional behavior change; performance optimization only. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
