[ https://issues.apache.org/jira/browse/SPARK-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14110233#comment-14110233 ]
Apache Spark commented on SPARK-3043: ------------------------------------- User 'jkbradley' has created a pull request for this issue: https://github.com/apache/spark/pull/2125 > DecisionTree aggregation is inefficient > --------------------------------------- > > Key: SPARK-3043 > URL: https://issues.apache.org/jira/browse/SPARK-3043 > Project: Spark > Issue Type: Improvement > Components: MLlib > Affects Versions: 1.1.0 > Reporter: Joseph K. Bradley > Assignee: Joseph K. Bradley > > 2 major efficiency issues in computation and storage: > (1) DecisionTree aggregation involves reshaping data unnecessarily. > E.g., the internal methods extractNodeInfo() and getBinDataForNode() involve > reshaping the data multiple times without real computation. > (2) DecisionTree splits and aggregate bins can include many unused > bins/splits. > The same number of splits/bins are used for all features. E.g., if there is > a continuous feature which uses 100 bins, then there will also be 100 bins > allocated for all binary features, even though only 2 are necessary. -- This message was sent by Atlassian JIRA (v6.2#6252) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org