Re: mllib model in production web API

Hi Vincent I am not sure whether you are asking me or Nicolas. If me, then no we didn't. Never used Akka and wasn't even aware that it has such capabilities. Using Java API so we don't have Akka as a dependency right now. On Tue, Oct 18, 2016 at 12:47 PM, vincent gromakowski <

Re: mllib model in production web API

Hi Did you try applying the model with akka instead of spark ? https://spark-summit.org/eu-2015/events/real-time-anomaly-detection-with-spark-ml-and-akka/ Le 18 oct. 2016 5:58 AM, "Aseem Bansal" a écrit : > @Nicolas > > No, ours is different. We required predictions within

Re: mllib model in production web API

@Nicolas No, ours is different. We required predictions within 10ms time frame so we needed much less latency than that. Every algorithm has some parameters. Correct? We took the parameters from the mllib and used them to create ml package's model. ml package's model's prediction time was much

Re: mllib model in production web API

Hi Sean and Aseem, thanks both. A simple thing which sped things up greatly was simply to load our sql (for one record effectively) directly and then convert to a dataframe, rather than using Spark to load it. Sounds stupid, but this took us from > 5 seconds to ~1 second on a very small instance.

Re: mllib model in production web API

Hi Faced a similar issue. Our solution was to load the model, cache it after converting it to a model from mllib and then use that instead of ml model. On Tue, Oct 11, 2016 at 10:22 PM, Sean Owen wrote: > I don't believe it will ever scale to spin up a whole distributed job

Re: mllib model in production web API

I don't believe it will ever scale to spin up a whole distributed job to serve one request. You can look possibly at the bits in mllib-local. You might do well to export as something like PMML either with Spark's export or JPMML and then load it into a web container and score it, without Spark

mllib model in production web API

Hi all, so I have a model which has been stored in S3. And I have a Scala webapp which for certain requests loads the model and transforms submitted data against it. I'm not sure how to run this quickly on a single instance though. At the moment Spark is being bundled up with the web app in an