flyrain commented on code in PR #11130:
URL: https://github.com/apache/iceberg/pull/11130#discussion_r1777877117
##########
format/spec.md:
##########
@@ -298,16 +298,143 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
+| Field id, name | Type | Description
|
+|----------------------------------|---------------|---------------------------------------------------------------------------------------------------------|
+| **`2147483646 _file`** | `string` | Path of the file in which
a row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row
in the source data file, starting at `0` |
+| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
+| **`2147483643 _spec_id`** | `int` | Spec ID used to track the
file containing a row |
+| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files
|
+| **`2147483545 pos`** | `long` | Ordinal position of a
row, used in position-based delete files
|
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used
in position-based delete files |
+| **`2147483543 _row_id`** | `long` | A unique long assigned
when row-lineage is enabled see [Row Lineage](#row-lineage)
|
+| **`2147483542 _last_update`** | `long` | The sequence number which
last updated this row when row-lineage is enabled [Row Lineage](#row-lineage) |
+
+### Row Lineage
+
+In Specification V3, an Iceberg Table can declare that engines must track
row-lineage of all newly created rows. This
+requirement is controlled by setting the field `row-lineage` to true in the
table's metadata. When true, two additional
+fields in data files will be available for all rows added to the table.
+
+* `_row_id` a unique long for every row. Computed via inheritance for rows in
their original datafiles
Review Comment:
Nit: How about "_row_id is a unique long as an identifier for every row
within a table."?
This version clearly communicates that the uniqueness applies to each row in
the context of a specific table, not a snapshot, not across-tables. I know it's
subtle, but the change itself is minor. May worth to consider.
##########
format/spec.md:
##########
@@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
+| Field id, name | Type | Description
|
+|----------------------------------|---------------|---------------------------------------------------------------------------------------------------------|
+| **`2147483646 _file`** | `string` | Path of the file in which
a row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row
in the source data file, starting at `0` |
+| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
+| **`2147483643 _spec_id`** | `int` | Spec ID used to track the
file containing a row |
+| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files
|
+| **`2147483545 pos`** | `long` | Ordinal position of a
row, used in position-based delete files
|
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used
in position-based delete files |
+| **`2147483543 _row_id`** | `long` | A unique long assigned
when row-lineage is enabled see [Row Lineage](#row-lineage)
|
+| **`2147483542 _last_update`** | `long` | The sequence number which
last updated this row when row-lineage is enabled [Row Lineage](#row-lineage) |
+
+### Row Lineage
+
+In v3 and later, an Iceberg table can track row lineage fields for all newly
created rows. Row lineage is enabled by setting the field `row-lineage` to
true in the table's metadata. When enabled, engines must maintain the
`next-row-id` table field and the following row-level fields when writing data
files:
+
+* `_row_id` a unique long for every row. Assigned via inheritance when a row
is added and the existing value is explicitly written when the row is written
to a new file.
+* `_last_update` the sequence number of the commit that last updated a row.
The value is inherited when a row is added or modified and the existing value
is explicitly written when the row is written to a different data file but not
modified.
+
+These fields are assigned and updated by inheritance because the commit
sequence number and starting row ID are not assigned until the snapshot is
successfully committed. Inheritance is used to allow writing data and manifest
files before values are known so that it is not necessary to rewrite data and
manifest files when an optimistic commit is retried.
Review Comment:
We may need to rewrite manifest list file in case of retrying. As the
following examples shows, the manifest file `add1`, `add2` and `add3` may have
different `first-row-id` in case of retrying. Should we mention it here?
##########
format/spec.md:
##########
@@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
+| Field id, name | Type | Description
|
+|----------------------------------|---------------|---------------------------------------------------------------------------------------------------------|
+| **`2147483646 _file`** | `string` | Path of the file in which
a row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row
in the source data file, starting at `0` |
+| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
+| **`2147483643 _spec_id`** | `int` | Spec ID used to track the
file containing a row |
+| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files
|
+| **`2147483545 pos`** | `long` | Ordinal position of a
row, used in position-based delete files
|
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used
in position-based delete files |
+| **`2147483543 _row_id`** | `long` | A unique long assigned
when row-lineage is enabled see [Row Lineage](#row-lineage)
|
+| **`2147483542 _last_update`** | `long` | The sequence number which
last updated this row when row-lineage is enabled [Row Lineage](#row-lineage) |
+
+### Row Lineage
+
+In v3 and later, an Iceberg table can track row lineage fields for all newly
created rows. Row lineage is enabled by setting the field `row-lineage` to
true in the table's metadata. When enabled, engines must maintain the
`next-row-id` table field and the following row-level fields when writing data
files:
+
+* `_row_id` a unique long for every row. Assigned via inheritance when a row
is added and the existing value is explicitly written when the row is written
to a new file.
+* `_last_update` the sequence number of the commit that last updated a row.
The value is inherited when a row is added or modified and the existing value
is explicitly written when the row is written to a different data file but not
modified.
+
+These fields are assigned and updated by inheritance because the commit
sequence number and starting row ID are not assigned until the snapshot is
successfully committed. Inheritance is used to allow writing data and manifest
files before values are known so that it is not necessary to rewrite data and
manifest files when an optimistic commit is retried.
+
+When row lineage is enabled, new snapshots cannot include [Equality
Deletes](#equality-delete-files). Row lineage is incompatible with equality
deletes because lineage values must be maintained, but equality deletes are
used to avoid reading existing data before writing changes.
+
+
+#### Row lineage assignment
+
+Row lineage fields are written when row lineage is enabled. When not enabled,
row lineage fields (`_row_id` and `_last_update`) must not be written to data
files. The rest of this section applies when row lineage is enabled.
Review Comment:
The same clean-up question here. Do we rewrite data files in case of
disabling row lineage or we disallow disabling?
##########
format/spec.md:
##########
@@ -298,16 +298,143 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
+| Field id, name | Type | Description
|
+|----------------------------------|---------------|---------------------------------------------------------------------------------------------------------|
+| **`2147483646 _file`** | `string` | Path of the file in which
a row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row
in the source data file, starting at `0` |
+| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
+| **`2147483643 _spec_id`** | `int` | Spec ID used to track the
file containing a row |
+| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files
|
+| **`2147483545 pos`** | `long` | Ordinal position of a
row, used in position-based delete files
|
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used
in position-based delete files |
+| **`2147483543 _row_id`** | `long` | A unique long assigned
when row-lineage is enabled see [Row Lineage](#row-lineage)
|
+| **`2147483542 _last_update`** | `long` | The sequence number which
last updated this row when row-lineage is enabled [Row Lineage](#row-lineage) |
Review Comment:
+1 for the name suggestion
##########
format/spec.md:
##########
@@ -298,16 +298,101 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
+| Field id, name | Type | Description
|
+|----------------------------------|---------------|---------------------------------------------------------------------------------------------------------|
+| **`2147483646 _file`** | `string` | Path of the file in which
a row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row
in the source data file, starting at `0` |
+| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
+| **`2147483643 _spec_id`** | `int` | Spec ID used to track the
file containing a row |
+| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files
|
+| **`2147483545 pos`** | `long` | Ordinal position of a
row, used in position-based delete files
|
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used
in position-based delete files |
+| **`2147483543 _row_id`** | `long` | A unique long assigned
when row-lineage is enabled see [Row Lineage](#row-lineage)
|
+| **`2147483542 _last_update`** | `long` | The sequence number which
last updated this row when row-lineage is enabled [Row Lineage](#row-lineage) |
+
+### Row Lineage
+
+In v3 and later, an Iceberg table can track row lineage fields for all newly
created rows. Row lineage is enabled by setting the field `row-lineage` to
true in the table's metadata. When enabled, engines must maintain the
`next-row-id` table field and the following row-level fields when writing data
files:
+
+* `_row_id` a unique long for every row. Assigned via inheritance when a row
is added and the existing value is explicitly written when the row is written
to a new file.
+* `_last_update` the sequence number of the commit that last updated a row.
The value is inherited when a row is added or modified and the existing value
is explicitly written when the row is written to a different data file but not
modified.
+
+These fields are assigned and updated by inheritance because the commit
sequence number and starting row ID are not assigned until the snapshot is
successfully committed. Inheritance is used to allow writing data and manifest
files before values are known so that it is not necessary to rewrite data and
manifest files when an optimistic commit is retried.
+
+When row lineage is enabled, new snapshots cannot include [Equality
Deletes](#equality-delete-files). Row lineage is incompatible with equality
deletes because lineage values must be maintained, but equality deletes are
used to avoid reading existing data before writing changes.
+
+
+#### Row lineage assignment
+
+Row lineage fields are written when row lineage is enabled. When not enabled,
row lineage fields (`_row_id` and `_last_update`) must not be written to data
files. The rest of this section applies when row lineage is enabled.
+
+When a row is added or modified, the `_last_update` field is set to `null` so
that it is inherited when reading. Similarly, the `_row_id` field for an added
row is set to `null` and assigned when reading.
+
+On read, if `_last_update` is `null` it is assigned the `sequence_number` of
the data file's manifest entry. The data sequence number of a data file is
documented in [Sequence Number Inheritance](#sequence-number-inheritance).
+
+When `null`, a row's `_row_id` field is assigned to the `start_row_id` from
its containing data file plus the row position in that data file (`_pos`). A
data file's `start_row_id` field is assigned using inheritance and is
documented in [First Row ID Inheritance](#first-row-id-inheritance). A
manifest's `start_row_id` is assigned when writing the manifest list for a
snapshot and is documented in [First Row ID
Assignment](#first-row-id-assignment). A snapshot's `start-row-id` is to the
table's `next-row-id` and is documented in [Snapshot Row
IDs](#snapshot-row-ids).
Review Comment:
Can we be consistent with the name `first_row_id` and `start_row_id`?
##########
format/spec.md:
##########
@@ -298,16 +298,143 @@ Iceberg tables must not use field ids greater than
2147483447 (`Integer.MAX_VALU
The set of metadata columns is:
-| Field id, name | Type | Description |
-|-----------------------------|---------------|-------------|
-| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
-| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
-| **`2147483644 _deleted`** | `boolean` | Whether the row has been
deleted |
-| **`2147483643 _spec_id`** | `int` | Spec ID used to track the file
containing a row |
-| **`2147483642 _partition`** | `struct` | Partition to which a row
belongs |
-| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
-| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
-| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
Review Comment:
nit: avoid reformatting here?
##########
format/spec.md:
##########
@@ -598,6 +702,14 @@ Notes:
1. Lower and upper bounds are serialized to bytes using the single-object
serialization in Appendix D. The type of used to encode the value is the type
of the partition field data.
2. If -0.0 is a value of the partition field, the `lower_bound` must not be
+0.0, and if +0.0 is a value of the partition field, the `upper_bound` must not
be -0.0.
+#### First Row ID Assignment
+
+Row ID inheritance is used when row lineage is enabled. When not enabled, a
manifest's `first_row_id` must always be set to `null`. The rest of this
section applies when row lineage is enabled.
Review Comment:
Do we allow to disable row lineage for a table? If it is allowed, should we
rewrite the manifest files and data files when we disable it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
