Thanks Soumya - I guess the next step from here is to move the MLlib model from the Spark application with simply does the training, and giving to the client application which simply does the predictions. I will try the Kryo library to physically serialize the object and trade it across machines / applications.
Rather than writing it to file, I will send it over the network - any thoughts on that? Thanks! On Mon, Jul 14, 2014 at 1:43 PM, Soumya Simanta <soumya.sima...@gmail.com> wrote: > Please look at the following. > > https://github.com/ooyala/spark-jobserver > http://en.wikipedia.org/wiki/Predictive_Model_Markup_Language > https://github.com/EsotericSoftware/kryo > > You can train your model convert it to PMML and return that to your client > OR > > You can train your model and write that model (serialized object) to the > file system (local, HDFS, S3 etc) or a datastore and return a location back > to the client on a successful write. > > > > > > On Mon, Jul 14, 2014 at 4:27 PM, Aris Vlasakakis <a...@vlasakakis.com> > wrote: > >> Hello Spark community, >> >> I would like to write an application in Scala that i a model server. It >> should have an MLlib Linear Regression model that is already trained on >> some big set of data, and then is able to repeatedly call >> myLinearRegressionModel.predict() many times and return the result. >> >> Now, I want this client application to submit a job to Spark and tell the >> Spark cluster job to >> >> 1) train its particular MLlib model, which produces a LinearRegression >> model, and then >> >> 2) take the produced Scala >> org.apache.spark.mllib.regression.LinearRegressionModel *object*, serialize >> that object, and return this serialized object over the wire to my calling >> application. >> >> 3) My client application receives the serialized Scala (model) object, >> and can call .predict() on it over and over. >> >> I am separating the heavy lifting of training the model and doing model >> predictions; the client application will only do predictions using the >> MLlib model it received from the Spark application. >> >> The confusion I have is that I only know how to "submit jobs to Spark" by >> using the bin/spark-submit script, and then the only output I receive is >> stdout (as in, text). I want my scala appliction to hopefully submit the >> spark model-training programmatically, and for the Spark application to >> return a SERIALIZED MLLIB OBJECT, not just some stdout text! >> >> How can I do this? I think my use case of separating long-running jobs to >> Spark and using it's libraries in another application should be a pretty >> common design pattern. >> >> Thanks! >> >> -- >> Άρης Βλασακάκης >> Aris Vlasakakis >> > >
