[ https://issues.apache.org/jira/browse/SPARK-22925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-22925. ---------------------------------- Resolution: Incomplete > ml model persistence creates a lot of small files > ------------------------------------------------- > > Key: SPARK-22925 > URL: https://issues.apache.org/jira/browse/SPARK-22925 > Project: Spark > Issue Type: Improvement > Components: MLlib > Affects Versions: 2.1.2, 2.2.1, 2.3.0 > Reporter: Felix Cheung > Priority: Major > Labels: bulk-closed > > Today in when calling model.save(), some ML models we do makeRDD(data, 1) or > repartition(1) but in some other models we don't. > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60 > In the former case issues such as SPARK-19294 have been reported for making > very large single file. > Whereas in the latter case, models such as RandomForestModel could create > hundreds or thousands of very small files which is also unmanageable. Looking > into this, there is no simple way to set/change spark.default.parallelism > (which would be pick up by sc.parallelize) while the app is running since > SparkConf seems to be copied/cached by the backend without a way to update > them. > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443 > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155 > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135 > It seems we need to have a way to make numSlice settable on a per-use basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org