stevenzwu commented on code in PR #16936:
URL: https://github.com/apache/iceberg/pull/16936#discussion_r3954102999
##########
api/src/main/java/org/apache/iceberg/ManifestFile.java:
##########
@@ -210,6 +229,16 @@ default Long firstRowId() {
return null;
}
+ /** Returns the number of entries in the manifest file, or null for pre-v4
manifests. */
+ default Long recordCount() {
Review Comment:
Dropped `recordCount()` from `ManifestFile`.
##########
api/src/main/java/org/apache/iceberg/ManifestFile.java:
##########
@@ -186,6 +189,22 @@ default boolean hasDeletedFiles() {
/** Returns the total number of rows in all files with status DELETED in the
manifest file. */
Long deletedRowsCount();
+ /**
+ * Returns the number of files with status REPLACED in the manifest file, or
null for pre-v4
+ * manifests.
+ */
+ default Integer replacedFilesCount() {
+ return null;
Review Comment:
Defaults now return 0 / 0L.
##########
core/src/main/java/org/apache/iceberg/GenericManifestFile.java:
##########
@@ -60,6 +60,8 @@ public class GenericManifestFile extends
SupportsIndexProjection
private PartitionFieldSummary[] partitions = null;
private byte[] keyMetadata = null;
private Long firstRowId = null;
+ private Long recordCount = null;
+ private int formatVersion = LEGACY_FORMAT_VERSION;
Review Comment:
Dropped the overrides; interface defaults cover it.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -53,6 +63,49 @@ static DeleteFile asEqualityDeleteFile(TrackedFile file,
Map<Integer, PartitionS
return new TrackedEqualityDeleteFile(file, resolveSpec(file, specsById));
}
+ /**
+ * Returns a reusable wrapper that presents a {@link DataFile} as a {@link
TrackedFile} row.
+ *
+ * @param formatVersion the target table's format version (must be 4+)
Review Comment:
Dropped.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -53,6 +63,49 @@ static DeleteFile asEqualityDeleteFile(TrackedFile file,
Map<Integer, PartitionS
return new TrackedEqualityDeleteFile(file, resolveSpec(file, specsById));
}
+ /**
+ * Returns a reusable wrapper that presents a {@link DataFile} as a {@link
TrackedFile} row.
+ *
+ * @param formatVersion the target table's format version (must be 4+)
+ * @param tableSchema table schema used to build {@link ContentStats} from
the file's stats
+ * @param metricsConfig metrics config used to prune the content stats schema
Review Comment:
Write wrappers now take a single `TrackedFile` write schema.
##########
api/src/main/java/org/apache/iceberg/ManifestFile.java:
##########
@@ -29,6 +29,9 @@
public interface ManifestFile {
int PARTITION_SUMMARIES_ELEMENT_ID = 508;
+ /** Format version for pre-v4 manifest files. */
+ int LEGACY_FORMAT_VERSION = 0;
Review Comment:
Removed. Pre-v4 now uses the `formatVersion()` default of 0.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -53,6 +63,49 @@ static DeleteFile asEqualityDeleteFile(TrackedFile file,
Map<Integer, PartitionS
return new TrackedEqualityDeleteFile(file, resolveSpec(file, specsById));
}
+ /**
+ * Returns a reusable wrapper that presents a {@link DataFile} as a {@link
TrackedFile} row.
+ *
+ * @param formatVersion the target table's format version (must be 4+)
+ * @param tableSchema table schema used to build {@link ContentStats} from
the file's stats
+ * @param metricsConfig metrics config used to prune the content stats schema
+ * @param partitionType target partition struct type; use one spec's
partition type for a
+ * single-spec manifest, or the union across live specs for a multi-spec
manifest
+ */
+ static DataTrackedFile forDataFile(
Review Comment:
Unwrapped already-adapted files. Write wrappers stay reusable for the writer
hot path.
##########
core/src/main/java/org/apache/iceberg/TableMetadata.java:
##########
@@ -59,6 +59,7 @@ public class TableMetadata implements Serializable {
static final int MIN_FORMAT_VERSION_ROW_LINEAGE = 3;
static final int MIN_FORMAT_VERSION_PARQUET_MANIFESTS = 4;
static final int MIN_FORMAT_VERSION_OPTIONAL_LOCATION = 4;
+ static final int MIN_FORMAT_VERSION_ADAPTIVE_MANIFEST_TREE = 4;
Review Comment:
Reused `MIN_FORMAT_VERSION_PARQUET_MANIFESTS`.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -53,6 +63,49 @@ static DeleteFile asEqualityDeleteFile(TrackedFile file,
Map<Integer, PartitionS
return new TrackedEqualityDeleteFile(file, resolveSpec(file, specsById));
}
+ /**
+ * Returns a reusable wrapper that presents a {@link DataFile} as a {@link
TrackedFile} row.
+ *
+ * @param formatVersion the target table's format version (must be 4+)
+ * @param tableSchema table schema used to build {@link ContentStats} from
the file's stats
+ * @param metricsConfig metrics config used to prune the content stats schema
+ * @param partitionType target partition struct type; use one spec's
partition type for a
+ * single-spec manifest, or the union across live specs for a multi-spec
manifest
Review Comment:
Wrappers derive partition and content-stats types from that schema.
##########
core/src/main/java/org/apache/iceberg/GenericManifestFile.java:
##########
@@ -130,6 +132,50 @@ public GenericManifestFile(Schema avroSchema) {
this.partitions = partitions == null ? null : partitions.toArray(new
PartitionFieldSummary[0]);
this.keyMetadata = ByteBuffers.toByteArray(keyMetadata);
this.firstRowId = firstRowId;
+ this.recordCount = null;
+ this.formatVersion = LEGACY_FORMAT_VERSION;
+ }
+
+ /** v4+ constructor variant that accepts recordCount and formatVersion. */
+ GenericManifestFile(
Review Comment:
Dropped the v4 fields/constructor from `GenericManifestFile`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
