stevenzwu commented on code in PR #14234:
URL: https://github.com/apache/iceberg/pull/14234#discussion_r3126076221
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
Review Comment:
> Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`
This intro starts with `struct<struct<...>>`, which is a bit abstract. is
this clearer?
The statistics for each field are tracked by a struct of counts, bounds,
sizes (described in [the later section](...)). Each field-level statistics
struct is a field of the content_stats struct, which holds all statistics for
table fields. Logically, content_stats therefore has the shape
struct<struct<...>>.
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
+Its **field id** is deterministically calculated as defined in the previous
section.
+
+###### Field stats types
+
+Each stats struct holds statistics for one table column. It may contain the
following metrics:
Review Comment:
The table has hints ("Only included for float/double types", "for
variable-length types") scattered across descriptions. A single applicability
column would be clearer, e.g.:
- value_count: all types
- null_value_count: optional columns only
- nan_value_count: float, double only
- avg_value_size_in_bytes, max_value_size_in_bytes: string, binary only (and
presumably variant? fixed? -- spell it out)
- lower_bound, upper_bound: all types, with type-specific rules
- exact_bounds: only for types where bounds may be truncated
This also gives writers a single place to check compliance.
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
+Its **field id** is deterministically calculated as defined in the previous
section.
+
+###### Field stats types
+
+Each stats struct holds statistics for one table column. It may contain the
following metrics:
+
+| required/optional | Offset | Name | Type |
Description
|
+|-------------------|--------|-------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| _optional_ | 1 | value_count | `long` |
Number of values in the column (including null and NaN values)
|
+| _optional_ | 2 | null_value_count | `long` |
Number of null values in the column. Only included for optional columns
|
+| _optional_ | 3 | nan_value_count | `long` |
Number of NaN values in the column. Only included for float/double types. NaN
rules follow note 2 under [Data File Fields](#data-file-fields)
|
+| _optional_ | 4 | avg_value_size_in_bytes | `int` |
Avg stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 5 | max_value_size_in_bytes | `int` |
Max stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 6 | lower_bound | type of table field |
Lower bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 7 | upper_bound | type of table field |
Upper bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 8 | exact_bounds | `boolean` |
Whether the `lower_bound` / `upper_bound` are exact (`true`) or may be
truncated or otherwise inexact (`false`). Defaults to `true`. Types such as
`string` / `binary` often use `false` when bounds are truncated. For types with
inherently exact bounds when written (for example boolean, integer,
floating-point, date, time, timestamp, decimal, uuid, `geometry`, `geography`),
writers should use `true` when bounds are present. If a deletion vector or
equality delete file can match rows in the data file, implementations must
treat bounds as inexact for pruning (`exact_bounds` as `false`) |
+
+###### Stats projection
+
+To retrieve stats for a particular table field ID, one would always project by
stats ID, where the stats ID for a given table field ID can be calculated by
applying the reverse calculation.
+For data columns the reverse calculation would be:
+
+`table_field_id = (stats_struct_id - 10_000) / 200`
+
+For [reserved field IDs](#reserved-field-ids), the reverse calculation would
be:
+
+`table_field_id = stats_struct_id - num_reserved_field_ids +
(Integer.MAX_VALUE - stats_struct_id) + (stats_struct_id -
stats_space_field_id_start_for_metadata_fields) /
num_supported_stats_per_column`
+
+using `num_reserved_field_ids = 200`,
`stats_space_field_id_start_for_metadata_fields = 2_147_000_000`, and
`num_supported_stats_per_column = 200` (see [ID assignment for stats
fields](#id-assignment-for-stats-fields)).
+
+Below are examples for some table field ID -> stats struct id calculations.
+
+| Table Field ID | Stats ID of Stats struct |
+|---------------------|---------------------------|
+| 0 | 10_000 |
+| 1 | 10_200 |
+| 2 | 10_400 |
+| 5 | 11_000 |
+| 100 | 30_000 |
+| 1_000_000 | 200_010_000 |
+
+| Reserved Field ID | Stats ID of Stats struct |
+|---------------------|---------------------------|
+| 2_147_483_447 | 2_147_000_000 |
+| 2_147_483_448 | 2_147_000_200 |
+| 2_147_483_541 | 2_147_018_800 |
+| 2_147_483_645 | 2_147_039_600 |
+| 2_147_483_646 | 2_147_039_800 |
+
+The below table shows the stats IDs of individual field statistics, which are
calculated based on the offset that is described in the [Field stats types
section](#field-stats-types)
+
+| Table Field ID | Stats ID of Stats struct | Stats Type | Stats
ID of individual statistic |
+|----------------|--------------------------|-------------------------|----------------------------------|
+| 2 | 10_400 | value_count | 10_401
|
+| | | null_value_count | 10_402
|
+| | | nan_value_count | 10_403
|
+| | | avg_value_size_in_bytes | 10_404
|
+| | | max_value_size_in_bytes | 10_405
|
+| | | lower_bound | 10_406
|
+| | | upper_bound | 10_407
|
+| | | exact_bounds | 10_408
|
+| 5 | 11_000 | value_count | 11_001
|
+| | | null_value_count | 11_002
|
+| | | nan_value_count | 11_003
|
+| | | avg_value_size_in_bytes | 11_004
|
+| | | max_value_size_in_bytes | 11_005
|
+| | | lower_bound | 11_006
|
+| | | upper_bound | 11_007
|
+| | | exact_bounds | 11_008
|
+
+###### Manifest schema and `content_stats` typing
Review Comment:
we only talked type promotion for schema evolution. should we also spell out
other cases?
- Add column: when a new column is added, the schema of content_stats grows
by one field. What about manifests written before the addition -- readers must
tolerate the missing field, right? State that.
- Drop column: does the dropped column's stats struct remain in newer
manifests' content_stats type? (Iceberg typically retains dropped-column
entries in old data, but new manifests?) Spell it out.
- Rename column: the content_stats struct field name (the numeric ID) is
stable, so rename is a no-op on the stats side. Worth stating.
##########
format/spec.md:
##########
@@ -683,6 +683,13 @@ Notes:
5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
6. The following field ids are reserved on `data_file`: 141.
+###### File-level column statistics
+
+Per-column metrics used for filtering and planning are stored at **file**
granularity on the `data_file` struct.
+In v3, implementations use maps such as `value_counts`, `lower_bounds`, and
`upper_bounds`, keyed by column id, with bounds serialized as binary (see note
1 under [Data File Fields](#data-file-fields)).
+Iceberg v4 adds the optional `content_stats` struct, which holds the same
*logical* metrics for primitive leaf columns using nested structs and typed
bounds (see [Content Stats](#content-stats)).
Review Comment:
> holds the same *logical* metrics
This is not entirely accurate, as the `column_sizes` is not kept in v4.
Instead two new metrics (avg and max value size) are added in v4.
maybe we should drop this part and clearly describe that `column_sizes` are
deprecated in v4?
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
Review Comment:
Also wondering if this formula is a little simpler
```
stats_struct_id = 2_147_000_000 + 200 * (reserved_field_id - 2_147_483_447)
```
##########
format/spec.md:
##########
@@ -683,6 +683,13 @@ Notes:
5. The `content_offset` and `content_size_in_bytes` fields are used to
reference a specific blob for direct access to a deletion vector. For deletion
vectors, these values are required and must exactly match the `offset` and
`length` stored in the Puffin footer for the deletion vector blob.
6. The following field ids are reserved on `data_file`: 141.
+###### File-level column statistics
+
+Per-column metrics used for filtering and planning are stored at **file**
granularity on the `data_file` struct.
+In v3, implementations use maps such as `value_counts`, `lower_bounds`, and
`upper_bounds`, keyed by column id, with bounds serialized as binary (see note
1 under [Data File Fields](#data-file-fields)).
+Iceberg v4 adds the optional `content_stats` struct, which holds the same
*logical* metrics for primitive leaf columns using nested structs and typed
bounds (see [Content Stats](#content-stats)).
Review Comment:
> for primitive leaf columns
maybe we can tighten up the wording a little bit.
```
For every primitive leaf field in the table schema, content_stats contains
one stats struct. Non-leaf fields (structs, lists, maps) do not get a stats
struct.
```
What about map keys and list elements? do they get stats structs too, like
v3 value_counts does via nested element/key/value field ids? maybe need to
spell it out.
For variant/geometry/geography leaves (which the bounds subsection already
covers), spell out whether they get a stats struct.
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
+Its **field id** is deterministically calculated as defined in the previous
section.
+
+###### Field stats types
+
+Each stats struct holds statistics for one table column. It may contain the
following metrics:
+
+| required/optional | Offset | Name | Type |
Description
|
+|-------------------|--------|-------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| _optional_ | 1 | value_count | `long` |
Number of values in the column (including null and NaN values)
|
+| _optional_ | 2 | null_value_count | `long` |
Number of null values in the column. Only included for optional columns
|
+| _optional_ | 3 | nan_value_count | `long` |
Number of NaN values in the column. Only included for float/double types. NaN
rules follow note 2 under [Data File Fields](#data-file-fields)
|
+| _optional_ | 4 | avg_value_size_in_bytes | `int` |
Avg stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 5 | max_value_size_in_bytes | `int` |
Max stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 6 | lower_bound | type of table field |
Lower bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 7 | upper_bound | type of table field |
Upper bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 8 | exact_bounds | `boolean` |
Whether the `lower_bound` / `upper_bound` are exact (`true`) or may be
truncated or otherwise inexact (`false`). Defaults to `true`. Types such as
`string` / `binary` often use `false` when bounds are truncated. For types with
inherently exact bounds when written (for example boolean, integer,
floating-point, date, time, timestamp, decimal, uuid, `geometry`, `geography`),
writers should use `true` when bounds are present. If a deletion vector or
equality delete file can match rows in the data file, implementations must
treat bounds as inexact for pruning (`exact_bounds` as `false`) |
+
+###### Stats projection
+
+To retrieve stats for a particular table field ID, one would always project by
stats ID, where the stats ID for a given table field ID can be calculated by
applying the reverse calculation.
+For data columns the reverse calculation would be:
+
+`table_field_id = (stats_struct_id - 10_000) / 200`
+
+For [reserved field IDs](#reserved-field-ids), the reverse calculation would
be:
+
+`table_field_id = stats_struct_id - num_reserved_field_ids +
(Integer.MAX_VALUE - stats_struct_id) + (stats_struct_id -
stats_space_field_id_start_for_metadata_fields) /
num_supported_stats_per_column`
Review Comment:
this formula can be simplified as?
```
reserved_field_id = 2_147_483_447 + (stats_struct_id - 2_147_000_000) / 200
```
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
Review Comment:
> The value 200 is both the width of each column's stats block and
num_reserved_field_ids from [Reserved field ids]
This coupling is incidental, not intentional. If we ever increase
num_supported_stats_per_column (to fit more future metrics), we should be free
to do so without touching num_reserved_field_ids. Please define them as two
independent constants with their own rationale ("200 ids per column leaves
headroom for ~190 future metrics") and drop the sentence equating them.
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
Review Comment:
Purely numeric field names are unusual. We probably should provide some
rationale for the name assignment.
- Why the name is numeric: because Iceberg's on-disk manifest reads project
by ID, the name is only a stable label.
- A normative statement for readers: readers MUST resolve content stats by
field ID, not by name; the name is informational.
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
+Its **field id** is deterministically calculated as defined in the previous
section.
+
+###### Field stats types
+
+Each stats struct holds statistics for one table column. It may contain the
following metrics:
+
+| required/optional | Offset | Name | Type |
Description
|
+|-------------------|--------|-------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| _optional_ | 1 | value_count | `long` |
Number of values in the column (including null and NaN values)
|
+| _optional_ | 2 | null_value_count | `long` |
Number of null values in the column. Only included for optional columns
|
+| _optional_ | 3 | nan_value_count | `long` |
Number of NaN values in the column. Only included for float/double types. NaN
rules follow note 2 under [Data File Fields](#data-file-fields)
|
+| _optional_ | 4 | avg_value_size_in_bytes | `int` |
Avg stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 5 | max_value_size_in_bytes | `int` |
Max stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
+| _optional_ | 6 | lower_bound | type of table field |
Lower bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 7 | upper_bound | type of table field |
Upper bound serialized as the column's type. Bounds follow rules defined in
[Bounds for Variant, Geometry, and
Geography](#bounds-for-variant-geometry-and-geography)
|
+| _optional_ | 8 | exact_bounds | `boolean` |
Whether the `lower_bound` / `upper_bound` are exact (`true`) or may be
truncated or otherwise inexact (`false`). Defaults to `true`. Types such as
`string` / `binary` often use `false` when bounds are truncated. For types with
inherently exact bounds when written (for example boolean, integer,
floating-point, date, time, timestamp, decimal, uuid, `geometry`, `geography`),
writers should use `true` when bounds are present. If a deletion vector or
equality delete file can match rows in the data file, implementations must
treat bounds as inexact for pruning (`exact_bounds` as `false`) |
Review Comment:
> or may be truncated or otherwise inexact (`false`)
or truncated (`false`)
> Defaults to `true`.
An optional field defaults to null, not a semantic value. Saying "Defaults
to true" is a bit confusing. maybe sth like this
```
Only written for types that may be truncated; absence means bounds are exact.
````
> If a deletion vector or equality delete file can match rows in the data
file, implementations must treat bounds as inexact for pruning (`exact_bounds`
as `false`)
This is a reader pruning rule, not a writer's field value. should the
description keep only what determines the written value (exact vs truncated)?
Should we write the reader pruning rule in a separate subsection?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
