Spark MLlib: Should I call .cache before fitting a model?

2018-02-27 Thread Gevorg Hari
Imagine that I am training a Spark MLlib model as follows: val traingData = loadTrainingData(...)val logisticRegression = new LogisticRegression() traingData.cacheval logisticRegressionModel = logisticRegression.fit(trainingData) Does the call traingData.cache improve performances at training

Spark MLlib Question - Online Scoring of PipelineModel

2018-01-05 Thread Gevorg Hari
Is Spark planning to support *online scoring* (without any Spark dependencies) of a PipelineModel trained offline? Not being able to do so is a huge barrier to entry for using Spark in production at my company... For online support, I found this https://github.com/combust/mleap Any feedback on