nastra commented on code in PR #14234:
URL: https://github.com/apache/iceberg/pull/14234#discussion_r3072620971
##########
format/spec.md:
##########
@@ -707,6 +714,119 @@ For `geography` only, xmin (X value of `lower_bounds`)
may be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+##### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>`. The statistics for fields are tracked inside a nested
struct of value counts and bounds (described in the next section). Each
field-level statistics struct is a field of the `content_stats` struct, which
holds all statistics for table fields.
+
+###### ID assignment for stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+The offset defined in the [field stats types section](#field-stats-types) is
added to the stats ID of the enclosing stats struct to calculate IDs for each
individual field stats type.
+
+**Data columns (normal table field ids)**
+
+Let `table_field_id` be the column's id in the table schema. Allocate a
contiguous block of **200** ids per column (`num_supported_stats_per_column =
200`). The stats struct for that column starts at:
+
+`stats_struct_id = 10_000 + (200 * table_field_id)`
+
+Each field statistic listed under [Field stats types](#field-stats-types) has
a fixed **offset** within that block. The field id for an individual field
statistic is:
+
+`stats_field_id = stats_struct_id + offset`
+
+The constant `10_000` is `stats_space_field_id_start_for_data_fields`. The
value **200** is both the width of each column's stats block and
`num_reserved_field_ids` from [Reserved field ids](#reserved-field-ids).
+
+**Reserved table field ids.**
+
+Columns whose ids fall in the [reserved field ID](#reserved-field-ids) space
use a different base so their stats ids do not overlap data columns:
+
+`stats_struct_id = 2_147_000_000 + (200 * (200 - (Integer.MAX_VALUE -
table_field_id)))`
+
+Here `2_147_000_000` is `stats_space_field_id_start_for_metadata_fields`. This
separate base is required because reserved ids are near `Integer.MAX_VALUE` and
cannot use the same linear mapping as data field ids.
+
+Valid data field ids support stats structs with ids from `10_000` through
`200_010_000`, so the highest supported **data** field id is `1_000_000`.
+
+###### Name assignment for `content_stats` fields
+
+Each nested stats struct is a **child field** of the root `content_stats`
struct. Its **name** is the numerical string of the table column's field id
(for example id `103` uses the name `"103"`).
+Its **field id** is deterministically calculated as defined in the previous
section.
+
+###### Field stats types
+
+Each stats struct holds statistics for one table column. It may contain the
following metrics:
+
+| required/optional | Offset | Name | Type |
Description
|
+|-------------------|--------|-------------------------|---------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| _optional_ | 1 | value_count | `long` |
Number of values in the column (including null and NaN values)
|
+| _optional_ | 2 | null_value_count | `long` |
Number of null values in the column. Only included for optional columns
|
+| _optional_ | 3 | nan_value_count | `long` |
Number of NaN values in the column. Only included for float/double types. NaN
rules follow note 2 under [Data File Fields](#data-file-fields)
|
+| _optional_ | 4 | avg_value_size_in_bytes | `int` |
Avg stored (compressed, encoded) value size in bytes for variable-length types
(`string` / `binary`)
|
Review Comment:
I think avg / max value sizes should represent the uncompressed / unencoded
size to have better estimates for CBO in Spark/Trino
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
