zhengruifeng opened a new pull request, #58328: URL: https://github.com/apache/spark/pull/58328
### What changes were proposed in this pull request? This is a follow-up to #57389. This PR changes `Node.withLeafIndices` to assign leaf indices directly on the existing leaf nodes and return the original root. It no longer rebuilds the complete `Node` graph. The leaf index is therefore a package-private mutable field that is initialized once before the model is published. The tests verify that leaf indexing preserves root identity and assigns the expected left-to-right indices. The old-API conversion tests now also call `predictLeaf` on converted classification and regression trees. ### Why are the changes needed? The implementation in #57389 constructed a second complete node graph while the original graph was still reachable. During training and model loading, that added O(number of nodes) temporary driver memory and could cause an out-of-memory failure for a large tree. Assigning the integer fields in place avoids replacement node allocations while preserving the same DFS ordering. Calling `predictLeaf` in the legacy conversion tests ensures those paths cannot omit leaf-index initialization unnoticed. ### Does this PR introduce _any_ user-facing change? No. This changes the internal initialization of an optimization on the unreleased `master` branch. Leaf IDs and prediction behavior remain unchanged. ### How was this patch tested? Ran: `JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 ./build/sbt 'mllib/testOnly org.apache.spark.ml.tree.impl.RandomForestSuite org.apache.spark.ml.classification.DecisionTreeClassifierSuite org.apache.spark.ml.regression.DecisionTreeRegressorSuite'` All 60 tests passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
