Re: best way to do deep learning on spark ?
In the meantime there is also deeplearning4j which integrates with Spark (for both Java and Scala): http://deeplearning4j.org/ Regards, James On 17 March 2016 at 02:32, Ulanov, Alexander wrote: > Hi Charles, > > > > There is an implementation of multilayer perceptron in Spark (since 1.5): > > > https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier > > > > Other features such as autoencoder, convolutional layers, etc. are > currently under development. Please refer to > https://issues.apache.org/jira/browse/SPARK-5575 > > > > Best regards, Alexander > > > > *From:* charles li [mailto:charles.up...@gmail.com] > *Sent:* Wednesday, March 16, 2016 7:01 PM > *To:* user > *Subject:* best way to do deep learning on spark ? > > > > > > Hi, guys, I'm new to MLlib on spark, after reading the document, it seems > that MLlib does not support deep learning, I want to know is there any way > to implement deep learning on spark ? > > > > *Do I must use 3-party package like caffe or tensorflow ?* > > > > or > > > > *Does deep learning module list in the MLlib development plan?* > > > > > great thanks > > > > -- > > *--* > > a spark lover, a quant, a developer and a good man. > > > > http://github.com/litaotao >
Re: best way to do deep learning on spark ?
Hi, Alexander, that's awesome, and when will that feature be released ? Since I want to know the opportunity cost between waiting for that release and use caffe or tensorFlow ? great thanks again On Thu, Mar 17, 2016 at 10:32 AM, Ulanov, Alexander < alexander.ula...@hpe.com> wrote: > Hi Charles, > > > > There is an implementation of multilayer perceptron in Spark (since 1.5): > > > https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier > > > > Other features such as autoencoder, convolutional layers, etc. are > currently under development. Please refer to > https://issues.apache.org/jira/browse/SPARK-5575 > > > > Best regards, Alexander > > > > *From:* charles li [mailto:charles.up...@gmail.com] > *Sent:* Wednesday, March 16, 2016 7:01 PM > *To:* user > *Subject:* best way to do deep learning on spark ? > > > > > > Hi, guys, I'm new to MLlib on spark, after reading the document, it seems > that MLlib does not support deep learning, I want to know is there any way > to implement deep learning on spark ? > > > > *Do I must use 3-party package like caffe or tensorflow ?* > > > > or > > > > *Does deep learning module list in the MLlib development plan?* > > > > > great thanks > > > > -- > > *--* > > a spark lover, a quant, a developer and a good man. > > > > http://github.com/litaotao > -- *--* a spark lover, a quant, a developer and a good man. http://github.com/litaotao
Re: best way to do deep learning on spark ?
On Thu, Mar 17, 2016 at 3:51 AM, charles li wrote: > Hi, Alexander, > > that's awesome, and when will that feature be released ? Since I want to > know the opportunity cost between waiting for that release and use caffe or > tensorFlow ? > I don't expect MLlib will be able to compete with major players like Caffe or TensorFlow, at least not in the short term. Check out https://github.com/amplab/SparkNet. It's from AMPLab (like Spark), and it runs Caffe or TensorFlow on Spark. I think it's the state of the art for deep learning on Spark. great thanks again > > On Thu, Mar 17, 2016 at 10:32 AM, Ulanov, Alexander < > alexander.ula...@hpe.com> wrote: > >> Hi Charles, >> >> >> >> There is an implementation of multilayer perceptron in Spark (since 1.5): >> >> >> https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier >> >> >> >> Other features such as autoencoder, convolutional layers, etc. are >> currently under development. Please refer to >> https://issues.apache.org/jira/browse/SPARK-5575 >> >> >> >> Best regards, Alexander >> >> >> >> *From:* charles li [mailto:charles.up...@gmail.com] >> *Sent:* Wednesday, March 16, 2016 7:01 PM >> *To:* user >> *Subject:* best way to do deep learning on spark ? >> >> >> >> >> >> Hi, guys, I'm new to MLlib on spark, after reading the document, it seems >> that MLlib does not support deep learning, I want to know is there any way >> to implement deep learning on spark ? >> >> >> >> *Do I must use 3-party package like caffe or tensorflow ?* >> >> >> >> or >> >> >> >> *Does deep learning module list in the MLlib development plan?* >> >> >> >> >> great thanks >> >> >> >> -- >> >> *--* >> >> a spark lover, a quant, a developer and a good man. >> >> >> >> http://github.com/litaotao >> > > > > -- > *--* > a spark lover, a quant, a developer and a good man. > > http://github.com/litaotao >
RE: best way to do deep learning on spark ?
Hi Charles, There is an implementation of multilayer perceptron in Spark (since 1.5): https://spark.apache.org/docs/latest/ml-classification-regression.html#multilayer-perceptron-classifier Other features such as autoencoder, convolutional layers, etc. are currently under development. Please refer to https://issues.apache.org/jira/browse/SPARK-5575 Best regards, Alexander From: charles li [mailto:charles.up...@gmail.com] Sent: Wednesday, March 16, 2016 7:01 PM To: user Subject: best way to do deep learning on spark ? Hi, guys, I'm new to MLlib on spark, after reading the document, it seems that MLlib does not support deep learning, I want to know is there any way to implement deep learning on spark ? Do I must use 3-party package like caffe or tensorflow ? or Does deep learning module list in the MLlib development plan? great thanks -- -- a spark lover, a quant, a developer and a good man. http://github.com/litaotao