uros-b commented on code in PR #17705:
URL: https://github.com/apache/iceberg/pull/17705#discussion_r3801600005
##########
site/docs/blog/posts/2026-08-12-variant-in-apache-iceberg.md:
##########
@@ -68,10 +72,32 @@ optional group payload (VARIANT(1)) {
}
```
-- `metadata` holds a dictionary of the field names used in the value, so the
`value` bytes reference each name by an integer id instead of repeating the
name string.
+- `metadata` holds the dictionary of field names used in the value, so those
names are not repeated inline with the data.
- `value` holds the encoded data: a scalar, an array, or an object. Arrays and
objects store a `field_offset` per element (the byte offset where that
element's value starts), and objects also store a `field_id` per field (an
index into the metadata dictionary).
-The Variant column itself is addressed by field ID like any other Iceberg
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+The Variant column itself is addressed by its Iceberg field ID like any other
column, but its `metadata` and `value` subfields are accessed by name, which
matters for shredding.
+
+The same Variant maps into every file format Iceberg supports: a Parquet
`group`, an Avro `record`, or an ORC `struct`, each holding the `metadata` and
`value` pair. In Avro and ORC, a Variant is always the single unshredded pair.
+
+### One column, many layouts
+
+A Variant's structure is not consistent across rows or files, but the column's
Iceberg type is always `variant`, whatever shapes flow through it. Adding or
removing a field inside the data changes only the bytes in each row.
+
+That one logical column can be laid out differently in each data file. In
Parquet, one file may store it unshredded as the `metadata` + `value` pair
while another shreds its hot fields into dedicated typed columns. Both files
carry the same Variant field ID, and a reader reconciles whichever layout it
finds:
+
+```text
+payload (one Variant column, one field ID)
+├─ data file A, unshredded: metadata + value
+└─ data file B, shredded: metadata + value + typed_value.event,
typed_value.country
+```
+
+Snapshots do not change this. Each snapshot records the schema that was
current when it was written, and because the Variant column keeps its field ID
across schema changes, time travel reads every file back through the same
column.
+
+### Statistics and data skipping
+
+Because Variant is a column in the Iceberg schema, the table's manifests carry
statistics for it, and that is what lets Iceberg skip files during planning.
Iceberg records value and null counts for a Variant column. When a field is
shredded into its own typed column, Iceberg also records lower and upper bounds
for it, stored as a Variant object whose keys are normalized JSON paths to each
field. An unshredded `value` blob is opaque, so it contributes counts but no
bounds.
Review Comment:
The spec contradicts the final sentence. The "Bounds for Variant" section of
format/spec.md defines a general mechanism (valid for shredded and unshredded
Variant data), where the Variant column's manifest lower_bounds/upper_bounds
entries are stored as Variant objects whose keys are normalized JSON path
expressions (e.g. $['event_type']) and whose values are per-field min/max
primitives. The spec explicitly states: "Including bounds for any field is
optional", it does not restrict bounds to files written with shredding.
Suggested fix: Change the paragraph to reflect the practical picture, e.g.
```
For an unshredded Variant column, computing per-field bounds requires
reading the raw value bytes, so writers typically record only counts. When a
field is shredded into its own typed column, its Parquet statistics are
available directly, and Iceberg can record lower and upper bounds for that
field path in the manifest, stored as a Variant object whose keys are
normalized JSON paths.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
