DeviousCardi opened a new pull request, #11550: URL: https://github.com/apache/arrow-rs/pull/11550
# Which issue does this PR close? - Closes #11520 # Rationale for this change A dictionary column whose keys are all valid but point at a null value has no physical nulls, so `RecordBatch::try_new` accepts it under a non-nullable field. `ArrowWriter` then writes a required Parquet column with no definition levels, so those rows read back as `0` (Int32) or `""` (Utf8) instead of null, with no error on write or read. `StructArray` and `ListArray` already reject such children because they check logical nulls, so in practice this only gets through at the top level of a batch. #11278 handles the same problem for REE arrays by making the Parquet column optional based on the REE values field. Dictionary values carry no field, and the Parquet schema is fixed before any data is seen, so the writer cannot make the column optional here. It returns an error instead. # What changes are included in this PR? `LevelInfoBuilder::try_new` returns an error for a dictionary leaf with no definition levels (the leaf and all its ancestors are non-nullable) that has logical nulls. The logical null buffer is already computed by `ArrayLevels::new`, so this adds no extra work. Leaves with a nullable ancestor are not checked, because a nullable parent may legitimately mask null values in a non-nullable child. # Are these changes tested? Yes. A new test in `parquet/tests/arrow_writer/roundtrip.rs` covers Int32 and Utf8 dictionaries where a valid key points at a null value: writing under a non-nullable field fails, and writing under a nullable field round-trips the nulls. It also checks that non-nullable dictionaries still write when there are no null values, or when a null value is never referenced. # Are there any significant user-facing changes? Writing such a batch now returns an error where it used to succeed and silently replace the nulls with values. --- This change was written with Claude Code (AI-generated: the fix, the test and this description). I reviewed the generated code and the description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
