zhengruifeng opened a new pull request, #58330:
URL: https://github.com/apache/spark/pull/58330
### What changes were proposed in this pull request?
This is an alternative follow-up to #57389. Unlike #58328, this approach
keeps
`LeafNode.leafIndex` immutable and assigns it when each leaf is first
constructed.
The three node-construction paths assign indices directly:
- `LearningNode.toNode` threads the next index through its left-first
recursion after accounting
for pruning.
- `Node.fromOld` threads the next index through legacy-model conversion.
- `buildTreeFromNodes` uses the persisted preorder node IDs: its reverse
traversal assigns leaf
indices from right to left, starting at `leafCount - 1`.
The post-construction `Node.withLeafIndices` copy is removed. Test fixtures
now provide explicit
leaf indices, tree equality checks compare them, and legacy conversion tests
exercise `predictLeaf`.
### Why are the changes needed?
The implementation in #57389 constructs an unindexed node graph and then
rebuilds the entire tree
to attach leaf indices. Although ensemble trees are processed sequentially,
this retains an extra
copy of the current tree and adds O(number of nodes in one tree) peak driver
memory during training,
loading, and legacy conversion.
Assigning indices during initial construction removes that temporary graph
while preserving
immutable model nodes and the existing left-to-right DFS leaf ordering.
### Does this PR introduce _any_ user-facing change?
No. This changes the internal initialization of an optimization on the
unreleased `master` branch.
Leaf IDs and prediction behavior remain unchanged.
### How was this patch tested?
Ran:
`JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 ./build/sbt 'mllib/testOnly
org.apache.spark.ml.tree.impl.RandomForestSuite
org.apache.spark.ml.classification.DecisionTreeClassifierSuite
org.apache.spark.ml.regression.DecisionTreeRegressorSuite'`
All 59 tests passed. `./dev/lint-scala` also passed Scalastyle and Scalafmt
checks.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
