amogh-jahagirdar commented on code in PR #6880:
URL: https://github.com/apache/iceberg/pull/6880#discussion_r1117118424
##########
delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java:
##########
@@ -213,6 +218,52 @@ private PartitionSpec
getPartitionSpecFromDeltaSnapshot(Schema schema) {
return builder.build();
}
+ /**
+ * Commit the initial delta snapshot to iceberg transaction. It tries the
snapshot starting from
+ * {@code deltaStartVersion} to {@code latestVersion} and commit the first
constructable one.
+ *
+ * <p>There are two cases that the delta snapshot is not constructable:
+ *
+ * <ul>
+ * <li>the version is earlier than the earliest checkpoint
+ * <li>the corresponding data files are deleted by {@code VACUUM}
+ * </ul>
+ *
+ * <p>For more information, please refer to delta lake's <a
+ * href="https://docs.delta.io/latest/delta-batch.html#-data-retention";>Data
Retention</a>
+ *
+ * @param latestVersion the latest version of the delta lake table
+ * @param transaction the iceberg transaction
+ * @return the initial version of the delta lake table that is successfully
committed to iceberg
+ */
+ private long commitInitialDeltaSnapshotToIcebergTransaction(
+ long latestVersion, Transaction transaction) {
+ long constructableStartVersion = deltaStartVersion;
+ while (constructableStartVersion <= latestVersion) {
+ try {
+ List<AddFile> initDataFiles =
+
deltaLog.getSnapshotForVersionAsOf(constructableStartVersion).getAllFiles();
+ List<DataFile> filesToAdd = Lists.newArrayList();
Review Comment:
Should `filesToAdd` be a set instead of a list, if the files could be
re-used across delta snapshots (I'm unfamiliar with the layout of the delta log
so maybe this is not the case)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
