Re: thought experiment: use spark ML to real time prediction

her similar use case. > > Kind regards > > Andy > > > > > From: Nick Pentreath <nick.pentre...@gmail.com> > Date: Wednesday, November 18, 2015 at 4:03 AM > To: DB Tsai <dbt...@dbtsai.com> > Cc: "user @spark" <user@spark.apache.org> &g

Re: thought experiment: use spark ML to real time prediction

;>> >>>>>> >>>>>> Sincerely, >>>>>> >>>>>> DB Tsai >>>>>> ------------------ >>>>>> Web: https://www.dbtsai.com >>>>>> PGP Key ID: 0xAF08DF8

Re: thought experiment: use spark ML to real time prediction

08DF8D >>>>> >>>>> On Thu, Nov 12, 2015 at 8:15 PM, Nirmal Fernando <nir...@wso2.com> >>>>> wrote: >>>>> >>>>>> >>>>>> >>>>>> On Fri, Nov 13, 2015 at 2:04 AM, darren <dar...@ontr

Re: thought experiment: use spark ML to real time prediction

> >>>>> On Thu, Nov 12, 2015 at 8:15 PM, Nirmal Fernando <nir...@wso2.com> wrote: >>>>>> >>>>>> >>>>>> On Fri, Nov 13, 2015 at 2:04 AM, darren <dar...@ontrenet.com> wrote: >>>>>>> >>>&g

Re: thought experiment: use spark ML to real time prediction

;>>> This is a very useful side effect of ML models. >>>>>> >>>>>> If mlib can't use models outside spark that's a real shame. >>>>>> >>>>> >>>>> Well you can as mentioned earlier. You don't need Spark

Re: thought experiment: use spark ML to real time prediction

gt; >>>> Original message >>>> From: "Kothuvatiparambil, Viju" < >>>> viju.kothuvatiparam...@bankofamerica.com> >>>> Date: 11/12/2015 3:09 PM (GMT-05:00) >>>> To: DB Tsai <dbt...@dbtsai.com>, S

Re: thought experiment: use spark ML to real time prediction

n as mentioned earlier. You don't need Spark runtime for >>>> predictions, save the serialized model and deserialize to use. (you need >>>> the Spark Jars in the classpath though) >>>> >>>>> >>>>> >>>>> Sent from my Verizo

RE: thought experiment: use spark ML to real time prediction

; Adrian Tanase; user @spark; Xiangrui Meng; hol...@pigscanfly.ca Subject: Re: thought experiment: use spark ML to real time prediction I think the use-case can be quick different from PMML. By having a Spark platform independent ML jar, this can empower users to do the following, 1) PMML doesn't

RE: thought experiment: use spark ML to real time prediction

Davidson <a...@santacruzintegration.com>, Adrian Tanase <atan...@adobe.com>, "user @spark" <user@spark.apache.org>, Xiangrui Meng <men...@gmail.com>, hol...@pigscanfly.ca Subject: RE: thought experiment: use spark ML to real time prediction I am glad to

Re: thought experiment: use spark ML to real time prediction

+1 on that. It would be useful to use the model outside of Spark. _ From: DB Tsai <dbt...@dbtsai.com> Sent: Wednesday, November 11, 2015 11:57 PM Subject: Re: thought experiment: use spark ML to real time prediction To: Nirmal Fernando <nir...@wso2.com&

Re: thought experiment: use spark ML to real time prediction

ote: > >> +1 on that. It would be useful to use the model outside of Spark. >> >> >> _ >> From: DB Tsai <dbt...@dbtsai.com> >> Sent: Wednesday, November 11, 2015 11:57 PM >> Subject: Re: thought experiment: use spar

Re: thought experiment: use spark ML to real time prediction

>, Sean Owen <so...@cloudera.com> >> Cc: Felix Cheung <felixcheun...@hotmail.com>, Nirmal Fernando < >> nir...@wso2.com>, Andy Davidson <a...@santacruzintegration.com>, Adrian >> Tanase <atan...@adobe.com>, "user @spark" <user@spark.apache.or

Re: thought experiment: use spark ML to real time prediction

rnando < > nir...@wso2.com>, Andy Davidson <a...@santacruzintegration.com>, Adrian > Tanase <atan...@adobe.com>, "user @spark" <user@spark.apache.org>, > Xiangrui Meng <men...@gmail.com>, hol...@pigscanfly.ca > Subject: RE: thought experiment: use

Re: thought experiment: use spark ML to real time prediction

be useful to use the model outside of Spark. >> >> >> _ >> From: DB Tsai <dbt...@dbtsai.com> >> Sent: Wednesday, November 11, 2015 11:57 PM >> Subject: Re: thought experiment: use spark ML to real time prediction >> To: Nirmal Fer

Re: thought experiment: use spark ML to real time prediction

As of now, we are basically serializing the ML model and then deserialize it for prediction at real time. On Wed, Nov 11, 2015 at 4:39 PM, Adrian Tanase wrote: > I don’t think this answers your question but here’s how you would evaluate > the model in realtime in a streaming

Re: thought experiment: use spark ML to real time prediction

I don’t think this answers your question but here’s how you would evaluate the model in realtime in a streaming app https://databricks.gitbooks.io/databricks-spark-reference-applications/content/twitter_classifier/predict.html Maybe you can find a way to extract portions of MLLib and run them

Re: thought experiment: use spark ML to real time prediction

Do you think it will be useful to separate those models and model loader/writer code into another spark-ml-common jar without any spark platform dependencies so users can load the models trained by Spark ML in their application and run the prediction? Sincerely, DB Tsai

RE: thought experiment: use spark ML to real time prediction

I have a similar issue. I want to load a model saved by a spark machine learning job, in a web application. model.save(jsc.sc(), "myModelPath"); LogisticRegressionModel model = LogisticRegressionModel.load(