errr...do you have any suggestions for me before 1.3 release? I can't believe there's no ML model serialize method in Spark. I think training the models are quite expensive, isn't it?
Thanks, David On Sun, Mar 8, 2015 at 5:14 AM Burak Yavuz <brk...@gmail.com> wrote: > Hi, > > There is model import/export for some of the ML algorithms on the current > master (and they'll be shipped with the 1.3 release). > > Burak > On Mar 7, 2015 4:17 AM, "Xi Shen" <davidshe...@gmail.com> wrote: > >> Wait...it seem SparkContext does not provide a way to save/load object >> files. It can only save/load RDD. What do I missed here? >> >> >> Thanks, >> David >> >> >> On Sat, Mar 7, 2015 at 11:05 PM Xi Shen <davidshe...@gmail.com> wrote: >> >>> Ah~it is serializable. Thanks! >>> >>> >>> On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy <ekremak...@gmail.com> >>> wrote: >>> >>>> You can serialize your trained model to persist somewhere. >>>> >>>> Ekrem Aksoy >>>> >>>> On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen <davidshe...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I checked a few ML algorithms in MLLib. >>>>> >>>>> https://spark.apache.org/docs/0.8.1/api/mllib/index.html# >>>>> org.apache.spark.mllib.classification.LogisticRegressionModel >>>>> >>>>> I could not find a way to save the trained model. Does this means I >>>>> have to train my model every time? Is there a more economic way to do >>>>> this? >>>>> >>>>> I am thinking about something like: >>>>> >>>>> model.run(...) >>>>> model.save("hdfs://path/to/hdfs") >>>>> >>>>> Then, next I can do: >>>>> >>>>> val model = Model.createFrom("hdfs://...") >>>>> model.predict(vector) >>>>> >>>>> I am new to spark, maybe there are other ways to persistent the model? >>>>> >>>>> >>>>> Thanks, >>>>> David >>>>> >>>>> >>>>
