HyukjinKwon opened a new pull request, #58596:
URL: https://github.com/apache/spark/pull/58596
### What changes were proposed in this pull request?
Two size computations that can overflow a 32-bit `Int` are made
overflow-safe:
- `ImageSchema.decode` computes the decoded byte size as `height * width *
nChannels`. This is now
computed in `Long` so the multiplication cannot wrap around and slip past
the existing
`< 1e9` size assertion (which would then allocate a wrongly-sized or
negative array).
- `Interaction` computes the output-vector size as a product of per-feature
output sizes with
`size *= currentEncoder.outputSize`. This now uses `Math.multiplyExact` so
that a product
exceeding `Int.MaxValue` fails with a clear arithmetic error instead of
wrapping to a bogus
(possibly negative) size and producing corrupt indices.
### Why are the changes needed?
Both products can exceed `Int.MaxValue` for large-but-representable inputs,
and silently wrapping
produces a wrong or negative allocation size rather than a clear failure. A
legitimate image
(< 1e9 bytes) and a legitimate feature interaction (which must fit in an
`Int`-indexed vector) are
unaffected.
### Does this PR introduce _any_ user-facing change?
No, other than a clear error instead of a corrupt result / wrong-size
allocation on inputs whose
size would overflow `Int`.
### How was this patch tested?
Existing `ImageSchema`/`Interaction` tests. The overflow branches are not
unit-tested because
triggering them requires inputs large enough (a multi-billion-pixel image;
nominal attribute
counts whose product exceeds `Int.MaxValue`) to exhaust test heap before the
check runs; the
changes are local (`Long` arithmetic / `Math.multiplyExact`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
