HyukjinKwon opened a new pull request, #58595: URL: https://github.com/apache/spark/pull/58595
### What changes were proposed in this pull request? When loading a saved `DecisionTreeModel` or `BisectingKMeansModel`, the node graph is rebuilt by following child node ids recursively. This adds cycle detection so that a set of node rows whose child ids form a cycle (which never happens for a model written by Spark) fails fast with a clear "Cycle detected ..." error instead of recursing until the driver hits a `StackOverflowError`. `BisectingKMeansModel` additionally memoizes already-built nodes, so a data set that references the same child id from many parents (a DAG) no longer expands exponentially. ### Why are the changes needed? Loading a model whose persisted node data is inconsistent should fail with a clear error rather than a `StackOverflowError` or an exponential blowup. Valid models have neither cycles nor shared children, so their loading is unchanged. ### Does this PR introduce _any_ user-facing change? No, other than a clearer error when loading a model with inconsistent node ids. ### How was this patch tested? Added tests to `DecisionTreeSuite` and `BisectingKMeansSuite` that save a real model, then load a copy whose node data has been made self-referential, and assert loading fails with "Cycle detected". ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
