rdblue commented on code in PR #14234:
URL: https://github.com/apache/iceberg/pull/14234#discussion_r3025182956
##########
format/spec.md:
##########
@@ -707,6 +707,92 @@ For `geography` only, xmin (X value of `lower_bounds`) may
be greater than xmax
When calculating upper and lower bounds for `geometry` and `geography`, null
or NaN values in a coordinate dimension are skipped; for example, POINT (1 NaN)
contributes a value to X but no values to Y, Z, or M dimension bounds. If a
dimension has only null or NaN values, that dimension is omitted from the
bounding box. If either the X or Y dimension is missing then the bounding box
itself is not produced.
+#### Content Stats
+
+Iceberg v4 introduces content stats which represent stats in a
`struct<struct<...>>` where each nested struct holds the stats for an
individual field of a table. The different field stats types are defined in the
next section.
+
+##### Field Stats Types
+
+The struct that holds individual stats for a particular field of a table
consists of the following fields:
+
+| Name | Type | Offset from field ID of base struct
| required | Description
|
+|------------------|---------------------|-------------------------------------|----------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| value_count | `long` | 1
| false | Number of values in the column (including null and NaN values)
|
+| null_value_count | `long` | 2
| false | Number of null values in the column
|
+| nan_value_count | `long` | 3
| false | Number of NaN values in the column
|
+| avg_value_count | `int` | 4
| false | The avg value count for variable-length types (string/binary)
|
+| max_value_count | `long` | 5
| false | The max value count for variable-length types (string/binary)
|
+| lower_bound | type of table field | 6
| false | Lower bound in the column serialized as the type of the column
itself. Each value must be less than or equal to all non-null, non-NaN values
in the column for the file [2] |
+| upper_bound | type of table field | 7
| false | Upper bound in the column serialized as the type of the column
itself. Each value must be greater than or equal to all non-null, non-NaN
values in the column for the file [2] |
+| exact_bounds | `boolean` | 8
| false | Whether the `upper_bound` / `lower_bound` is exact or not.
Defaults to true. Types such as string/binary can't have exact bounds.
Additionally, if a DV or an equality delete matches a given data file, then
`exact_bounds` must be treated as `false` |
+
+#### ID Assignment for Stats fields
+
+ID assignment follows a deterministic transform that maps from the **table ID
space** to the **metadata ID space**. For a given field ID from the **table ID
space** each nested stats struct gets an ID assigned from the **metadata ID
space**.
+Offsets defined in the [field stats types section](#field-stats-types) are
then applied to the stats ID of the enclosing stats struct to calculate IDs for
each individual field stats type.
+The calculation is: `stats_space_field_id_start_for_data_fields +
(num_supported_stats_per_column * table_field_id`)
+
+If the table field ID is a [reserved field ID](#reserved-field-ids) then a
slightly adjusted calculation is used:
`stats_space_field_id_start_for_metadata_fields +
(num_supported_stats_per_column * (num_reserved_field_ids - (Integer.MAX_VALUE
- metadata_field_id))`)
+
+The individual variables are defined as following:
+
+* `stats_space_field_id_start_for_data_fields = 10_000`
+* `stats_space_field_id_start_for_metadata_fields = 2_147_000_000`
+* `num_supported_stats_per_column = 200`, thus supporting a range of **200**
potential stats fields for a single table field
+* `num_reserved_field_ids = 200` as defined in [reserved field
ID](#reserved-field-ids)
+
+Note that stats field IDs are only calculated for an assigned field ID:
+
+* that is defined in the table field ID space
+* that is defined in the [reserved field ID](#reserved-field-ids) space
+
+The stats ID range is defined from `10_000` to `200_010_000`, which
effectively means that the highest supported data field ID is `1_000_000`.
+
+#### Stats projection
Review Comment:
You can think of the previous section as defining how to produce the type
for `content_stats`. I think this also requires a section on how to write
metadata files using those types. The `content_stats` type must be produced for
a table. That type is embedded in the manifest schema as some field ID. That is
the schema that is used to write metadata files for the table and to read
metadata files.
The reason why we need to specify that the schema is used for both writing
and reading is that this is how type promotion will work. An `int` field named
`x` will have a struct `x struct<value_count long, lower_bound int, upper_bound
int, ...>`. When it is promoted to `long`, the new struct will be `x
struct<value_count long, lower_bound long, upper_bound long>`, so that reading
metadata will use normal type promotion. Then when writing, the type that was
read will match the outgoing type (also the struct with `long` bounds).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
