just to give some reference point. with the same algorithm running on mnist dataset.
1. python implementation: ~10 miliseconds per iteration (can be faster if i switch to gpu) 2. local version (scala + breeze): ~2 seconds per iteration 3. distributed version (spark + scala + breeze): 15 seconds per iteration i love spark and really enjoy writing scala code. but this huge difference in performance makes it really hard to do any kind of machine learning work. On Thu, Dec 11, 2014 at 2:18 PM, Duy Huynh <duy.huynh....@gmail.com> wrote: > both. > > first, the distributed version is so much slower than python. i tried a > few things like broadcasting variables, replacing Seq with Array, and a few > other little things. it helps to improve the performance, but still slower > than the python code. > > so, i wrote a local version that's pretty much just running a bunch of > breeze/blas operations. i guess that's purely scala (no spark). this > local version is faster than the distributed version but still much slower > than the python code. > > > > > > > > On Thu, Dec 11, 2014 at 2:09 PM, Natu Lauchande <nlaucha...@gmail.com> > wrote: > >> Are you using Scala in a distributed enviroment or in a standalone mode ? >> >> Natu >> >> On Thu, Dec 11, 2014 at 8:23 PM, ll <duy.huynh....@gmail.com> wrote: >> >>> hi.. i'm converting some of my machine learning python code into scala + >>> spark. i haven't been able to run it on large dataset yet, but on small >>> datasets (like http://yann.lecun.com/exdb/mnist/), my spark + scala >>> code is >>> much slower than my python code (5 to 10 times slower than python) >>> >>> i already tried everything to improve my spark + scala code like >>> broadcasting variables, caching the RDD, replacing all my matrix/vector >>> operations with breeze/blas, etc. i saw some improvements, but it's >>> still a >>> lot slower than my python code. >>> >>> why is that? >>> >>> how do you improve your spark + scala performance today? >>> >>> or is spark + scala just not the right tool for small to medium datasets? >>> >>> when would you use spark + scala vs. python? >>> >>> thanks! >>> >>> >>> >>> -- >>> View this message in context: >>> http://apache-spark-user-list.1001560.n3.nabble.com/why-is-spark-scala-code-so-slow-compared-to-python-tp20636.html >>> Sent from the Apache Spark User List mailing list archive at Nabble.com. >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >>> For additional commands, e-mail: user-h...@spark.apache.org >>> >>> >> >
