nastra commented on code in PR #14234:
URL: https://github.com/apache/iceberg/pull/14234#discussion_r3071618858
##########
format/spec.md:
##########
@@ -707,6 +707,92 @@ For `geography` only, xmin (X value of `lower_bounds`) may
be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+#### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>` where each nested struct holds the stats for an
individual field of a table. The different field stats types are defined in the
next section.
+
+##### Field Stats Types
+
+The struct that holds individual stats for a particular field of a table
consists of the following fields:
+
+| Name | Type | Offset from field ID of base struct
| required | Description
|
+|------------------|---------------------|-------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| value_count | `long` | 1
| false | Number of values in the column (including null and NaN values)
|
+| null_value_count | `long` | 2
| false | Number of null values in the column
|
+| nan_value_count | `long` | 3
| false | Number of NaN values in the column
|
+| avg_value_count | `int` | 4
| false | The avg value count for variable-length types (string/binary)
|
+| max_value_count | `long` | 5
| false | The max value count for variable-length types (string/binary)
|
+| lower_bound | type of table field | 6
| false | Lower bound in the column serialized as the type of the column
itself. Each value must be less than or equal to all non-null, non-NaN values
in the column for the file [2] |
+| upper_bound | type of table field | 7
| false | Upper bound in the column serialized as the type of the column
itself. Each value must be greater than or equal to all non-null, non-NaN
values in the column for the file [2] |
+| exact_bounds | `boolean` | 8
| false | Whether the `upper_bound` / `lower_bound` is exact or not.
Defaults to true. Types such as string/binary can't have exact bounds.
Additionally, if a DV or an equality delete matches a given data file, then
`exact_bounds` must be treated as `false` |
+
+#### ID Assignment for Stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+Offsets defined in the [field stats types section](#field-stats-types) are
then applied to the stats ID of the enclosing stats struct to calculate IDs for
each individual field stats type.
+The calculation is: `stats_space_field_id_start_for_data_fields +
(num_supported_stats_per_column * table_field_id`)
+
+If the table field ID is a [reserved field ID](#reserved-field-ids) then a
slightly adjusted calculation is used:
`stats_space_field_id_start_for_metadata_fields +
(num_supported_stats_per_column * (num_reserved_field_ids - (Integer.MAX_VALUE
- metadata_field_id))`)
+
+The individual variables are defined as following:
+
+* `stats_space_field_id_start_for_data_fields = 10_000`
+* `stats_space_field_id_start_for_metadata_fields = 2_147_000_000`
+* `num_supported_stats_per_column = 200`, thus supporting a range of **200**
potential stats fields for a single table field
+* `num_reserved_field_ids = 200` as defined in [reserved field
ID](#reserved-field-ids)
+
+Note that stats field IDs are only calculated for an assigned field ID:
+
+* that is defined in the table field ID space
+* that is defined in the [reserved field ID](#reserved-field-ids) space
+
+The stats ID range is defined from `10_000` to `200_010_000`, which
effectively means that the highest supported data field ID is `1_000_000`.
+
+#### Stats projection
Review Comment:
makes sense, I've added a section
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
