Hello - I have a question on how to handle incremental model updation in Spark ML ..
We have a time series where we predict the future conditioned on the past. We can train a model offline based on historical data and then use that model during prediction. But say, if the underlying process is non-stationary, the probability distribution changes with time. In such cases we need to update the model so as to reflect the current change in distribution. We have 2 options - a. retrain periodically b. update the model incrementally a. is expensive. What about b. ? I think in Spark we have StreamingKMeans that takes care of this incremental model update for this classifier. Is this true ? But what about other classifiers that don't have these Streaming counterparts ? How do we handle such incremental model changes with those classifiers where the underlying distribution changes ? regards. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Incremental-model-update-tp27800.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
