Many ML algorithms are sequential because they were not designed to be
parallel. However, ML is not driven by algorithms in practice, but by
data and applications. As datasets getting bigger and bigger, some
algorithms got revised to work in parallel, like SGD and matrix
factorization. MLlib tries to implement those scalable algorithms that
can handle large-scale datasets.
That being said, even with sequential ML algorithms, Spark is helpful.
Because in practice we need to test multiple sets of parameters and
select the best one. Though the algorithm is sequential, the training
part is embarrassingly parallel. We can broadcast the whole dataset,
and then train model 1 on node 1, model 2 on node 2, etc. Cross
validation also falls into this category.
-Xiangrui
On Mon, Nov 3, 2014 at 1:55 PM, ll duy.huynh@gmail.com wrote:
i'm struggling with implementing a few algorithms with spark. hope to get
help from the community.
most of the machine learning algorithms today are sequential, while spark
is all about parallelism. it seems to me that using spark doesn't
actually help much, because in most cases you can't really paralellize a
sequential algorithm.
there must be some strong reasons why mllib was created and so many people
claim spark is ideal for machine learning.
what are those reasons?
what are some specific examples when how to use spark to implement
sequential machine learning algorithms?
any commen/feedback/answer is much appreciated.
thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/is-spark-a-good-fit-for-sequential-machine-learning-algorithms-tp18000.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
