amogh-jahagirdar commented on code in PR #16025: URL: https://github.com/apache/iceberg/pull/16025#discussion_r3935112023
########## format/spec.md: ########## @@ -706,9 +722,111 @@ The data and file sequence numbers are inherited only if the entry status is 1 ( Notes: -1. Technically, data files can be deleted when the last snapshot that contains the file as “live” data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots. +1. Technically, data files can be deleted when the last snapshot that contains the file as "live" data is garbage collected. But this is harder to detect and requires finding the diff of multiple snapshots. It is easier to track what files are deleted in a snapshot and delete them when that snapshot expires. It is not recommended to add a deleted file back to a table. Adding a deleted file can lead to edge cases where incremental deletes can break table snapshots. 2. Manifest list files are required in v2, so that the `sequence_number` and `snapshot_id` to inherit are always available. +=== "v4" + **Content Entries** + + | Field id | Name | Type | Required | Description | + |----------|------|------|----------|-------------| + | 134 | **`content_type`** | `int` (0: DATA, 3: DATA_MANIFEST, 4: DELETE_MANIFEST) | *required* | Type of content stored in the entry. Content types 3 and 4 are only valid in root manifests. | + | 157 | **`format_version`** | `int` (0: PRE-V4, 4: V4) | *required* | Writer format version. v4 writers must produce `format_version` 4. | + | 100 | **`location`** | `string` | *required* | Location of the file or manifest. | + | 101 | **`file_format`** | `string` | *required* | String file format name: `avro`, `orc`, `parquet`, or `puffin` | + | 147 | **`tracking`** | `tracking` struct | *required* | Groups status, snapshot, and sequence number. See tracking struct below. | + | 141 | **`spec_id`** | `int` | *optional* | ID of the partition spec used to write this manifest or data file. | + | 140 | **`sort_order_id`** | `int` | *optional* | ID representing sort order for this file. | + | 103 | **`record_count`** | `long` | *required* | Number of records in this file. | + | 104 | **`file_size_in_bytes`** | `long` | *required* | Total file size in bytes. | + | 146 | **`content_stats`** | `content_stats` struct | *optional* | Column stats. See [Column Stats Improvements](#column-stats-improvements). | + | 150 | **`manifest_info`** | `manifest_info` struct | *optional* | See manifest_info struct below. | + | 131 | **`key_metadata`** | `binary` | *optional* | Implementation-specific key metadata for encryption. | + | 132 | **`split_offsets`** | `list<133: long>` | *optional* | Split offsets for the data file. Must be sorted ascending. | + | 148 | **`deletion_vector`** | `deletion_vector` struct | *optional* | Row-level deletion vector for a data file. | + | 158 | **`column_files`** | `list<159: column_file>` | *optional* | Column update files associated with this entry. | + + Value 1 (POSITION_DELETES) is not used in v4. Writers must not produce `content_type` 1. + + Writers must not produce `content_type` 2 (EQUALITY DELETES) in v4. + + v4 leaf manifests must only contain entries with `content_type` 0 (DATA). A root manifest may reference v1-v3 manifests; v1-v3 leaf manifest references must have `format_version` set to 0. + + The following constraints apply based on `content_type`: + + - `manifest_info` must be set when `content_type` is 3 or 4; must be null otherwise. + - `deletion_vector` may only be set when `content_type` is 0; must be null otherwise. + - `column_files` must be null when `content_type` is not 0 or 3. + - `sort_order_id` must be null when `content_type` is 3 or 4 (manifests). + - `split_offsets` must be null when `content_type` is 3 or 4 (manifests). + - `tracking.deleted_positions` and `tracking.replaced_positions` must be null when `content_type` is not 3 or 4. + + **`tracking` struct (field 147)** + + | Field id | Name | Type | Required | Description | + |----------|------|------|----------|-------------| + | 0 | **`status`** | `int` (0: EXISTING, 1: ADDED, 2: DELETED, 3: REPLACED, 4: MODIFIED) | *required* | Used to track additions, deletions, replacements, and modifications. When a data file's `deletion_vector` or `column_files` change, REPLACED marks the prior version of the entry and MODIFIED marks the new, live version. For leaf manifest entries, MODIFIED marks a live manifest whose `dv` changed. Deletes are not used in scans. | + | 1 | **`snapshot_id`** | `long` | *optional* | Snapshot ID where the file was added or deleted. Inherited when null. Optional for leaf manifests, required for root. | + | 5 | **`dv_snapshot_id`** | `long` | *optional* | Snapshot ID where the deletion vector was added. Must be null when `deletion_vector` is null and `dv` is null. | + | 160 | **`latest_column_file_snapshot_id`** | `long` | *optional* | Snapshot ID where the latest column file was added. Inherited when null. Must be null when `column_files` is null. | + | 3 | **`sequence_number`** | `long` | *optional* | Data sequence number of the file. Inherited when null and status is 1 (ADDED). Must equal `file_sequence_number` if `content_type` is 3 or 4. Optional for leaf manifests, required for root. | + | 4 | **`file_sequence_number`** | `long` | *optional* | File sequence number indicating when the file was added. Inherited when null and status is ADDED. Must equal `sequence_number` if `content_type` is 3 or 4. | + | 142 | **`first_row_id`** | `long` | *optional* | The `_row_id` for the first row in the data file if `content_type` is 0. If `content_type` is 3, this is the starting `_row_id` to assign to rows added by ADDED data files. See [First Row ID Inheritance](#first-row-id-inheritance). | + | 6 | **`deleted_positions`** | `binary` | *optional* | Positions deleted in the referenced leaf manifest this snapshot. See [Manifest Deletion Vectors](#manifest-deletion-vectors). | + | 7 | **`replaced_positions`** | `binary` | *optional* | Positions replaced in the referenced leaf manifest this snapshot. See [Manifest Deletion Vectors](#manifest-deletion-vectors). | + + **`deletion_vector` struct (field 148)** + + | Field id | Name | Type | Required | Description | + |----------|------|------|----------|-------------| + | 155 | **`location`** | `string` | *required* | Location of the Puffin file. | + | 144 | **`offset`** | `long` | *required* | Offset in the file where the content starts. | + | 145 | **`size_in_bytes`** | `long` | *required* | Length of the referenced content stored in the file. | + | 156 | **`cardinality`** | `long` | *required* | Cardinality of the deletion vector. | + | 149 | **`key_metadata`** | `binary` | *optional* | Implementation-specific key metadata for encryption. | + + **`manifest_info` struct (field 150)** + + | Field id | Name | Type | Required | Description | + |----------|------|------|----------|-------------| + | 504 | **`added_files_count`** | `long` | *required* | Count of entries with status ADDED in the manifest. | + | 505 | **`existing_files_count`** | `long` | *required* | Count of entries with status EXISTING in the manifest. | + | 506 | **`deleted_files_count`** | `long` | *required* | Count of entries with status DELETED in the manifest. | + | 520 | **`replaced_files_count`** | `long` | *required* | Count of entries with status REPLACED in the manifest. | + | 524 | **`modified_files_count`** | `long` | *required* | Count of entries with status MODIFIED in the manifest. | + | 512 | **`added_rows_count`** | `long` | *required* | Total number of rows in ADDED entries. | + | 513 | **`existing_rows_count`** | `long` | *required* | Total number of rows in EXISTING entries. | + | 514 | **`deleted_rows_count`** | `long` | *required* | Total number of rows in DELETED entries. | + | 521 | **`replaced_rows_count`** | `long` | *required* | Total number of rows in REPLACED entries. | + | 525 | **`modified_rows_count`** | `long` | *required* | Total number of rows in MODIFIED entries. | + | 516 | **`min_sequence_number`** | `long` | *required* | Minimum data sequence number of all live entries in the manifest. | + | 522 | **`dv`** | `binary` | *optional* | Positions in the referenced leaf manifest that are not live. See [Manifest Deletion Vectors](#manifest-deletion-vectors). | + | 523 | **`dv_cardinality`** | `long` | *optional* | Cardinality of the manifest deletion vector. Must be set when `dv` is non-null; must be null otherwise. | + + **`column_file` struct (element 159 of `column_files`, field 158)** + + | Field id | Name | Type | Required | Description | + |----------|------|------|----------|-------------| + | 161 | **`format_version`** | `int` | *required* | Format version of this column file. | + | 162 | **`field_ids`** | `list<163: int>` | *required* | Live field IDs stored in this column file. | + | 164 | **`location`** | `string` | *required* | Location of the column file. | + | 165 | **`file_format`** | `string` | *required* | String file format name: `avro`, `orc`, or `parquet`. | + | 166 | **`file_size_in_bytes`** | `long` | *required* | Total column file size in bytes. | + | 167 | **`key_metadata`** | `binary` | *optional* | Implementation-specific key metadata for encryption. | + | 168 | **`split_offsets`** | `list<169: long>` | *optional* | Split offsets for the column file. Must be sorted ascending. | + + When a file is added to the dataset, its content entry must set status to ADDED (1) and store the snapshot ID in which the file was added. + + When a data file's deletion vector or column files are updated, the writer must record two content entries for the file in the snapshot: a REPLACED (3) entry for the prior version and a MODIFIED (4) entry for the new, live version. Every MODIFIED data file entry must have a corresponding REPLACED entry. Review Comment: "Two content entries" is bad wording for the case where manifest DVs/replaced diff DV is used to mark a leaf manifest position deleted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
