HyukjinKwon opened a new pull request, #58597:
URL: https://github.com/apache/spark/pull/58597
### What changes were proposed in this pull request?
Adds a new static SQL config `spark.sql.ml.maxNumFeatures` (default `-1`,
disabled) that bounds a
feature/attribute/vector count when it is derived from data rather than
fixed by the caller:
- `MLUtils.computeNumFeatures`: the feature dimension inferred from a libsvm
file,
- `AttributeGroup.fromMetadata`: the number of attributes declared in ML
column metadata,
- `VectorUDT.deserialize`: the declared size of a sparse vector read from
storage (which becomes
the dense array length on densification).
The value is read through a small helper (`MLMaxNumFeatures`) that reads it
once per `SparkEnv` and
caches it, so the per-row check in `VectorUDT.deserialize` does no per-row
config lookup or
allocation. `AttributeGroup.fromMetadata` also now rejects a negative or
out-of-`Int`-range
`NUM_ATTRIBUTES` (which would otherwise truncate to a negative/wrong array
size).
### Why are the changes needed?
When a feature count is read from a data file or column metadata, a very
large value can drive an
excessive driver allocation. This lets operators optionally cap it. The
check is off by default, so
there is no behavior change unless it is set.
### Does this PR introduce _any_ user-facing change?
Yes: a new opt-in static conf `spark.sql.ml.maxNumFeatures`, default `-1`
(disabled), which
preserves the previous behavior. When set to a positive value, inputs whose
inferred/declared
feature count exceeds it are rejected. `AttributeGroup.fromMetadata` also
rejects a negative or
out-of-`Int`-range attribute count with a clear error instead of failing
later.
### How was this patch tested?
Added `MLMaxNumFeaturesSuite` (a session configured with the cap; asserts
libsvm inference,
attribute metadata, and both `VectorUDT`s reject over-cap inputs and accept
within-cap ones) and a
case in `AttributeGroupSuite` for the negative / out-of-range attribute
count.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
