Hi all, I see this in V4 spec: "null_value_count is only used for optional fields" (https://github.com/apache/iceberg/blob/89e2f887491c1b5fa9f8b9de81b3aa8b31fa6974/format/spec.md?plain=1#L838)
This only makes sense if required field never contains nulls. But that actually is not true. In table schema, we can have a struct field which is optional and has a child field that is required. In that case, when the parent struct is null, the required field values is treated as NULL too. For example, below works in Spark: CREATE TABLE t (i INT, s STRUCT<a: INT NOT NULL, b: STRING>) USING iceberg; INSERT INTO t VALUES (1, NULL); SELECT COUNT (*) WHERE s.a IS NULL; -- returns 1 Because of this, I don't think the query engine can optimize the execution plan and assume a field doesn't contain null when it's marked as required. It would need to rely on the null_value_count explicitly being 0. If you agree on this, then we should always record the null_value_count no matter if it's required or optionals field. Thoughts?
