Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/2435#discussion_r17943410
--- Diff:
mllib/src/main/scala/org/apache/spark/mllib/tree/DecisionTree.scala ---
@@ -582,42 +472,36 @@ object DecisionTree extends Serializable with Logging
{
* drastically reduce the communication overhead.
*/
- // Common calculations for multiple nested methods:
-
- // numNodes: Number of nodes in this (level of tree, group),
- // where nodes at deeper (larger) levels may be divided
into groups.
- val numNodes = Node.maxNodesInLevel(level) / numGroups
+ // numNodes: Number of nodes in this group
+ val numNodes = nodesForGroup.values.map(_.size).sum
logDebug("numNodes = " + numNodes)
+ // Create node index:
+ // groupNodeIndex(treeIndex)(node index in tree) = node index in
aggregate statistics
+ // groupNodeMap(treeIndex)(node index in tree) = node
+ var idx = 0
+ val mutableGroupNodeIndex = new mutable.HashMap[Int, Map[Int, Int]]()
--- End diff --
minor: Later we can try to use `spark.util.collection.OpenHashMap` instead
for memory and speed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
