Thank you, Feynman, this is helpful. The paper that I linked claims a big speedup with FFTs for large convolution size. Though as you mentioned there is no FFT transformer in Spark yet. Moreover, we would need a parallel version of FFTs to support batch computations. So it probably worth considering matrix-matrix multiplication for convolution optimization at least as a first version. It can also take advantage of data batches.
From: Feynman Liang [mailto:fli...@databricks.com] Sent: Wednesday, September 09, 2015 12:56 AM To: Ulanov, Alexander Cc: Ruslan Dautkhanov; Nick Pentreath; user; na...@yandex.ru Subject: Re: Spark ANN My 2 cents: * There is frequency domain processing available already (e.g. spark.ml<http://spark.ml> DCT transformer) but no FFT transformer yet because complex numbers are not currently a Spark SQL datatype * We shouldn't assume signals are even, so we need complex numbers to implement the FFT * I have not closely studied the relative performance tradeoffs, so please do let me know if there's a significant difference in practice On Tue, Sep 8, 2015 at 5:46 PM, Ulanov, Alexander <alexander.ula...@hpe.com<mailto:alexander.ula...@hpe.com>> wrote: That is an option too. Implementing convolutions with FFTs should be considered as well http://arxiv.org/pdf/1312.5851.pdf. From: Feynman Liang [mailto:fli...@databricks.com<mailto:fli...@databricks.com>] Sent: Tuesday, September 08, 2015 12:07 PM To: Ulanov, Alexander Cc: Ruslan Dautkhanov; Nick Pentreath; user; na...@yandex.ru<mailto:na...@yandex.ru> Subject: Re: Spark ANN Just wondering, why do we need tensors? Is the implementation of convnets using im2col (see here<http://cs231n.github.io/convolutional-networks/>) insufficient? On Tue, Sep 8, 2015 at 11:55 AM, Ulanov, Alexander <alexander.ula...@hpe.com<mailto:alexander.ula...@hpe.com>> wrote: Ruslan, thanks for including me in the discussion! Dropout and other features such as Autoencoder were implemented, but not merged yet in order to have room for improving the internal Layer API. For example, there is an ongoing work with convolutional layer that consumes/outputs 2D arrays. We’ll probably need to change the Layer’s input/output type to tensors. This will influence dropout which will need some refactoring to handle tensors too. Also, all new components should have ML pipeline public interface. There is an umbrella issue for deep learning in Spark https://issues.apache.org/jira/browse/SPARK-5575 which includes various features of Autoencoder, in particular https://issues.apache.org/jira/browse/SPARK-10408. You are very welcome to join and contribute since there is a lot of work to be done. Best regards, Alexander From: Ruslan Dautkhanov [mailto:dautkha...@gmail.com<mailto:dautkha...@gmail.com>] Sent: Monday, September 07, 2015 10:09 PM To: Feynman Liang Cc: Nick Pentreath; user; na...@yandex.ru<mailto:na...@yandex.ru> Subject: Re: Spark ANN Found a dropout commit from avulanov: https://github.com/avulanov/spark/commit/3f25e26d10ef8617e46e35953fe0ad1a178be69d It probably hasn't made its way to MLLib (yet?). -- Ruslan Dautkhanov On Mon, Sep 7, 2015 at 8:34 PM, Feynman Liang <fli...@databricks.com<mailto:fli...@databricks.com>> wrote: Unfortunately, not yet... Deep learning support (autoencoders, RBMs) is on the roadmap for 1.6<https://issues.apache.org/jira/browse/SPARK-10324> though, and there is a spark package<http://spark-packages.org/package/rakeshchalasani/MLlib-dropout> for dropout regularized logistic regression. On Mon, Sep 7, 2015 at 3:15 PM, Ruslan Dautkhanov <dautkha...@gmail.com<mailto:dautkha...@gmail.com>> wrote: Thanks! It does not look Spark ANN yet supports dropout/dropconnect or any other techniques that help avoiding overfitting? http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf https://cs.nyu.edu/~wanli/dropc/dropc.pdf ps. There is a small copy-paste typo in https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/ann/BreezeUtil.scala#L43 should read B&C :) -- Ruslan Dautkhanov On Mon, Sep 7, 2015 at 12:47 PM, Feynman Liang <fli...@databricks.com<mailto:fli...@databricks.com>> wrote: Backprop is used to compute the gradient here<https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala#L579-L584>, which is then optimized by SGD or LBFGS here<https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/ann/Layer.scala#L878> On Mon, Sep 7, 2015 at 11:24 AM, Nick Pentreath <nick.pentre...@gmail.com<mailto:nick.pentre...@gmail.com>> wrote: Haven't checked the actual code but that doc says "MLPC employes backpropagation for learning the model. .."? — Sent from Mailbox<https://www.dropbox.com/mailbox> On Mon, Sep 7, 2015 at 8:18 PM, Ruslan Dautkhanov <dautkha...@gmail.com<mailto:dautkha...@gmail.com>> wrote: http://people.apache.org/~pwendell/spark-releases/latest/ml-ann.html Implementation seems missing backpropagation? Was there is a good reason to omit BP? What are the drawbacks of a pure feedforward-only ANN? Thanks! -- Ruslan Dautkhanov
