[
https://issues.apache.org/jira/browse/SPARK-22925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Felix Cheung updated SPARK-22925:
---------------------------------
Description:
Today in when calling model.save(), some ML models we do makeRDD(data, 1) or
repartition(1) but in some other models we don't.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60
In the former case issues such as SPARK-19294 have been reported for making
very large single file.
Whereas in the latter case, models such as RandomForestModel could create
hundreds or thousands of very small files which is also unmanageable. Looking
into this, there is no simple way to set/change spark.default.parallelism
(which would be pick up by sc.parallelize) while the app is running since
SparkConf seems to be copied/cached by the backend without a way to update them.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135
It seems we need to have a way to make numSlice settable on a per-use basis.
was:
Today in when calling model.save(), some ML models we do makeRDD(data, 1) or
repartition(1) but in some other models we don't.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60
In the former case issues such as SPARK-19294 have been reported for making
very large single file.
Whereas in the latter case, models such as RandomForestModel could create
hundreds or thousands of very small files which is also unmanageable. Looking
into this, there is no simple way to set/change spark.default.parallelism
(which would be pick up by sc.parallelize) while the app is running since
SparkConf seems to be copied/cached by the backend without a way to update them.
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135
It seems we need to have a way to make it settable on a per-use basis.
> ml model persistence creates a lot of small files
> -------------------------------------------------
>
> Key: SPARK-22925
> URL: https://issues.apache.org/jira/browse/SPARK-22925
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 2.1.2, 2.2.1, 2.3.0
> Reporter: Felix Cheung
>
> Today in when calling model.save(), some ML models we do makeRDD(data, 1) or
> repartition(1) but in some other models we don't.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/regression/impl/GLMRegressionModel.scala#L60
> In the former case issues such as SPARK-19294 have been reported for making
> very large single file.
> Whereas in the latter case, models such as RandomForestModel could create
> hundreds or thousands of very small files which is also unmanageable. Looking
> into this, there is no simple way to set/change spark.default.parallelism
> (which would be pick up by sc.parallelize) while the app is running since
> SparkConf seems to be copied/cached by the backend without a way to update
> them.
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/tree/model/treeEnsembleModels.scala#L443
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/BisectingKMeansModel.scala#L155
> https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala#L135
> It seems we need to have a way to make numSlice settable on a per-use basis.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
