JonSnow1807 opened a new pull request, #12096: URL: https://github.com/apache/arrow-rs/pull/12096
# Which issue does this PR close? - Closes #11832. # Rationale for this change `RecordBatch::try_new` checks non-nullable fields with `null_count()`, which only counts the validity buffer. Arrays whose nulls are logical rather than physical (a dictionary whose values contain referenced nulls, a run-end array with null runs, a union with null children, `NullArray`) passed the check even though they hold nulls. # What changes are included in this PR? - The check uses `logical_null_count()` instead, with the same error message. It only runs for non-nullable fields. Primitive arrays reuse their null buffer, and dictionary, run-end and union arrays compute the mask in O(len). - Two places relied on the old behaviour and are fixed here: ten `arrow-ipc` test fixtures declared non-nullable fields over arrays with logical nulls (now nullable), and `arrow-avro` emitted an Avro `"type": "null"` record field as non-nullable, so reading such a file now fails the check (now nullable in `AvroField::field()`). Happy to split the Avro change into its own PR. # Are these changes tested? Yes. New tests in `record_batch.rs` reject a dictionary with a referenced null value, a run-end array with a null run and a `NullArray`, and still accept a dictionary whose null value no key references. The rejection test fails on main. `arrow-array`, `arrow-ipc`, `arrow-avro`, `arrow`, `parquet` (all features, 2195 tests), `arrow-select`, `arrow-cast`, `arrow-csv` and `arrow-json` pass. # Are there any user-facing changes? Yes: `RecordBatch::try_new` and `try_new_with_options` now reject non-nullable columns that contain logical nulls, so code that built such batches will get an error. `try_new_unchecked` is unchanged. AI usage: code drafted by both human and AI, and reviewed by human. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
