Joseph K. Bradley created SPARK-3043:
----------------------------------------
Summary: DecisionTree aggregation is inefficient
Key: SPARK-3043
URL: https://issues.apache.org/jira/browse/SPARK-3043
Project: Spark
Issue Type: Improvement
Components: MLlib
Reporter: Joseph K. Bradley
2 major efficiency issues in computation and storage:
(1) DecisionTree aggregation involves reshaping data unnecessarily.
E.g., the internal methods extractNodeInfo() and getBinDataForNode() involve
reshaping the data multiple times without real computation.
(2) DecisionTree splits and aggregate bins can include many unused bins/splits.
The same number of splits/bins are used for all features. E.g., if there is a
continuous feature which uses 100 bins, then there will also be 100 bins
allocated for all binary features, even though only 2 are necessary.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
