zhengruifeng opened a new pull request, #58558: URL: https://github.com/apache/spark/pull/58558
### What changes were proposed in this pull request? This PR reduces the transform closure size of `MultilayerPerceptronClassificationModel`. The column-expression prediction hooks snapshot only the layer sizes, weights, and optional thresholds needed by each requested output column. A small serializable scoring wrapper builds its feed-forward network state lazily on the executor, so the closures do not retain the complete Spark ML model or serialize the network's derived dense weight representation. Companion-object helpers perform probability conversion without retaining any model state. ### Why are the changes needed? The default probabilistic classifier hooks invoke bound model methods. Their UDF closures therefore retain the complete `MultilayerPerceptronClassificationModel` and its parameter graph even though scoring only needs the network layout and fitted weights. This adds avoidable driver memory pressure for long-lived Spark Connect servers. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The following checks passed: ``` build/sbt mllib/compile build/sbt 'mllib/testOnly org.apache.spark.ml.classification.MultilayerPerceptronClassifierSuite' ``` `MultilayerPerceptronClassifierSuite` ran 15 tests. No new tests were added because its existing `testPredictMethods` coverage exercises every combination of raw-prediction, probability, and prediction output columns, in addition to single-instance prediction and model persistence. A temporary local probe used a deterministic network with 128 input units, 64 hidden units, and 3 output units. It extracted each `ScalaUDF.function` from the analyzed transform plan and serialized it with Spark's closure serializer. The former bound-model hooks were recreated in a temporary subclass. "All" serializes the functions for raw-prediction, probability, and prediction together. | Model/output | Before | After | Reduction | |---|---:|---:|---:| | Raw prediction | 74,712 B | 70,236 B | 6.0% | | Probability | 74,719 B | 70,244 B | 6.0% | | Prediction | 74,649 B | 70,210 B | 5.9% | | All | 75,257 B | 70,476 B | 6.4% | ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
