lucio35 created SPARK-22751:
-------------------------------
Summary: Improve ML RandomForest shuffle performance
Key: SPARK-22751
URL: https://issues.apache.org/jira/browse/SPARK-22751
Project: Spark
Issue Type: Improvement
Components: ML
Affects Versions: 2.2.0
Reporter: lucio35
When I try to use ML Randomforest to train a classifier with dataset
news20.binary, which has 19,996 training examples and 1,355,191 features, i
found that shuffle write size( 51 GB ) of findSplitsBySorting is very large
compared with the small data size( 133.52 MB ). I think it is useful to replace
groupByKey by reduceByKey to improve shuffle performance.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
