zhengruifeng opened a new pull request, #58623: URL: https://github.com/apache/spark/pull/58623
### What changes were proposed in this pull request? This PR reduces the broadcast payload created by `GBTRegressionModel.transform`. Instead of broadcasting the complete model, the transform broadcasts only the tree root nodes and a cloned tree-weight array. Prediction uses a companion-object helper over that state, while leaf prediction uses the existing `TreeEnsembleModel.predictLeaf` helper. ### Why are the changes needed? Prediction and leaf traversal need the root nodes, and prediction additionally needs the tree weights. Broadcasting the complete model also serializes unrelated state such as the model and tree-model parameter graphs, UIDs, and metadata. Avoiding that state reduces driver and executor memory pressure for long-lived Spark Connect servers. A temporary local probe trained a 20-tree model with 30 total nodes and serialized each broadcast payload using Spark's configured serializer. The full model required 53,494 bytes, while the root nodes and tree weights required 3,284 bytes, a 93.9% reduction. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? The following checks passed: ``` build/sbt mllib/compile build/sbt 'mllib/testOnly org.apache.spark.ml.regression.GBTRegressorSuite' ``` `GBTRegressorSuite` ran 16 tests covering transform prediction and leaf-index output. No new test was added because the change only narrows the serialized state used by those existing code paths. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
