Yeah, I'd agree with Nick. To have an implementation of RNN/LSTM in Spark, you may need a comprehensive abstraction of neural networks which is general enough to represent the computation (think of Torch, Keras, Tensorflow, MXNet, Caffe, etc.), and modify current computation engine to work with various computing units such as GPU. I don't think we will have such thing to be in Spark in the near future.
There are many efforts to integrate Spark and the specialized frameworks doing well in this abstraction and parallel computation. The best approach I think is to look at this efforts and contribute to them if possible. Nick Pentreath wrote > The short answer is there is none and highly unlikely to be inside of > Spark > MLlib any time in the near future. > > The best bets are to look at other DL libraries - for JVM there is > Deeplearning4J and BigDL (there are others but these seem to be the most > comprehensive I have come across) - that run on Spark. Also there are > various flavours of TensorFlow / Caffe on Spark. And of course the libs > such as Torch, Keras, Tensorflow, MXNet, Caffe etc. Some of them have Java > or Scala APIs and some form of Spark integration out there in the > community > (in varying states of development). > > Integrations with Spark are a bit patchy currently but include the > "XOnSpark" flavours mentioned above and TensorFrames (again, there may be > others). > > On Thu, 23 Feb 2017 at 14:23 n1kt0 < > nikita.balyschew@ > > wrote: > >> Hi, >> can anyone tell me what the current status about RNNs in Spark is? >> >> >> >> -- >> View this message in context: >> http://apache-spark-developers-list.1001551.n3.nabble.com/Implementation-of-RNN-LSTM-in-Spark-tp14866p21060.html >> Sent from the Apache Spark Developers List mailing list archive at >> Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: > dev-unsubscribe@.apache >> >> ----- Liang-Chi Hsieh | @viirya Spark Technology Center http://www.spark.tc/ -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Implementation-of-RNN-LSTM-in-Spark-tp14866p21094.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
