llxgdtop opened a new pull request, #12124:
URL: https://github.com/apache/arrow-rs/pull/12124
# Which issue does this PR close?
- Closes #11848
# Rationale for this change
`StructArray::try_new_with_length` validated non-nullable fields by
materializing the child's logical null buffer before doing anything
else. A `NullArray` stores no data, so `NullArray::new(usize::MAX)`
costs nothing to build — but its logical null buffer is proportional
to the length, and the validation panics on a ~2.3 EB allocation
instead of returning an `InvalidArgumentError`.
# What changes are included in this PR?
- When the struct has no nulls of its own, check
`logical_null_count()` instead, which `NullArray` implements in O(1).
- Only materialize `logical_nulls()` for the bit-level `contains`
comparison when parent nulls are present.
- The check is otherwise equivalent: `contains()` already returns
`true` for a buffer with no nulls, so the separate `null_count > 0`
test is no longer needed.
# Are these changes tested?
Yes. A new regression test builds a `StructArray` with a non-nullable
field backed by `NullArray::new(usize::MAX)`. It panics with a 2.25 EB
allocation failure on `main`, and returns `InvalidArgumentError` with
this change. `cargo test -p arrow-array` (754 unit + 208 doc tests),
clippy `-D warnings` and `cargo fmt --check` all pass.
# Are there any significant user-facing changes?
No. The only behavioural difference is that this invalid input now
surfaces as an error instead of a panic.
---
This PR was prepared with AI assistance: I used it to help me
understand the validation code and the root cause, and to draft this
description. I understand the change and stand behind it.
All three failing jobs fail in environment setup steps (`Install HawkEye`,
Archery `Build`, `Setup Python`) — every code check passes.
The HawkEye one looks like a tooling issue rather than anything on this
branch: hawkeye 7.3.0 was released on Nov 2 and `install-action` picks up
the latest release. I installed the same version locally from crates.io
and ran `hawkeye check --fail-on-unknown`: 862 files, 0 conflicts, exit 0.
So the prebuilt 7.3.0 binaries may be the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
