[ https://issues.apache.org/jira/browse/SPARK-22925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Cheung updated SPARK-22925: --------------------------------- Issue Type: Improvement (was: Bug) > ml model persistence creates a lot of small files > ------------------------------------------------- > > Key: SPARK-22925 > URL: https://issues.apache.org/jira/browse/SPARK-22925 > Project: Spark > Issue Type: Improvement > Components: MLlib > Affects Versions: 2.1.2, 2.2.1, 2.3.0 > Reporter: Felix Cheung > > Today in when calling model.save(), some ML models we do makeRDD(data, 1) or > repartition(1) but in some other models we don't. > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60 > In the former case issues such as SPARK-19294 have been reported for making > very large single file. > Whereas in the latter case, models such as RandomForestModel could create > hundreds or thousands of files which is also unmanageable. Looking into this, > there is no simple way to set/change spark.default.parallelism (which would > be pick up by sc.parallelize) while the app is running since SparkConf seems > to be copied/cached by the backend without a way to update them. > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443 > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155 > https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135 > It seems we need to have a way to make it settable on a per-use basis. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org