Re: Issue in parallelization of CNN model using spark
Dear All Thanks all of you for your reply. I am trying to parallelize the CNN model using Keras2DML library, however, I am getting the error message: NO Module Named Systemml.mllearn. Can any body guide me how to install systemml using ubuntu best regards On Tue, Jul 14, 2020 at 4:34 AM Anwar AliKhan wrote: > This is very useful for me leading on from week4 of the Andrew Ng course. > > > On Mon, 13 Jul 2020, 15:18 Sean Owen, wrote: > >> There is a multilayer perceptron implementation in Spark ML, but >> that's not what you're looking for. >> To parallelize model training developed using standard libraries like >> Keras, use Horovod from Uber. >> https://horovod.readthedocs.io/en/stable/spark_include.html >> >> On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan wrote: >> > >> > Dear Spark User >> > >> > I am trying to parallelize the CNN (convolutional neural network) model >> using spark. I have developed the model using python and Keras library. The >> model works fine on a single machine but when we try on multiple machines, >> the execution time remains the same as sequential. >> > Could you please tell me that there is any built-in library for CNN to >> parallelize in spark framework. Moreover, MLLIB does not have any support >> for CNN. >> > Best regards >> > Mukhtaj >> > >> > >> > >> > >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >>
Re: Issue in parallelization of CNN model using spark
Ok, thanks. You can buy it here https://www.amazon.com/s?k=hands+on+machine+learning+with+scikit-learn+and+tensorflow+2&crid=2U0P9XVIJ790T&sprefix=Hands+on+machine+%2Caps%2C246&ref=nb_sb_ss_i_1_17 This book is like an accompaniment to the Andrew Ng course on coursera. It uses exact same mathematical notations , examples etc. so it is a smooth transition from that courses. On Tue, 14 Jul 2020, 15:52 Sean Owen, wrote: > It is still copyrighted material, no matter its state of editing. Yes, > you should not be sharing this on the internet. > > On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan > wrote: > > > > Please note It is freely available because it is an early unedited raw > edition. > > It is not 100% complete , it is not entirely same as yours. > > So it is not piracy. > > I agree it is a good book. > > >
Re: Issue in parallelization of CNN model using spark
It is still copyrighted material, no matter its state of editing. Yes, you should not be sharing this on the internet. On Tue, Jul 14, 2020 at 9:46 AM Anwar AliKhan wrote: > > Please note It is freely available because it is an early unedited raw > edition. > It is not 100% complete , it is not entirely same as yours. > So it is not piracy. > I agree it is a good book. > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Issue in parallelization of CNN model using spark
Please note It is freely available because it is an early unedited raw edition. It is not 100% complete , it is not entirely same as yours. So it is not piracy. I agree it is a good book. On Tue, 14 Jul 2020, 14:30 Patrick McCarthy, wrote: > Please don't advocate for piracy, this book is not freely available. > > I own it and it's wonderful, Mr. Géron deserves to benefit from it. > > On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan > wrote: > >> link to a free book which may be useful. >> >> Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow >> Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien >> Géron >> >> https://bit.ly/2zxueGt >> >> >> >> >> >> 13 Jul 2020, 15:18 Sean Owen, wrote: >> >>> There is a multilayer perceptron implementation in Spark ML, but >>> that's not what you're looking for. >>> To parallelize model training developed using standard libraries like >>> Keras, use Horovod from Uber. >>> https://horovod.readthedocs.io/en/stable/spark_include.html >>> >>> On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan >>> wrote: >>> > >>> > Dear Spark User >>> > >>> > I am trying to parallelize the CNN (convolutional neural network) >>> model using spark. I have developed the model using python and Keras >>> library. The model works fine on a single machine but when we try on >>> multiple machines, the execution time remains the same as sequential. >>> > Could you please tell me that there is any built-in library for CNN to >>> parallelize in spark framework. Moreover, MLLIB does not have any support >>> for CNN. >>> > Best regards >>> > Mukhtaj >>> > >>> > >>> > >>> > >>> >>> - >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> > > -- > > > *Patrick McCarthy * > > Senior Data Scientist, Machine Learning Engineering > > Dstillery > > 470 Park Ave South, 17th Floor, NYC 10016 >
Re: Issue in parallelization of CNN model using spark
Please don't advocate for piracy, this book is not freely available. I own it and it's wonderful, Mr. Géron deserves to benefit from it. On Mon, Jul 13, 2020 at 9:59 PM Anwar AliKhan wrote: > link to a free book which may be useful. > > Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow > Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien > Géron > > https://bit.ly/2zxueGt > > > > > > 13 Jul 2020, 15:18 Sean Owen, wrote: > >> There is a multilayer perceptron implementation in Spark ML, but >> that's not what you're looking for. >> To parallelize model training developed using standard libraries like >> Keras, use Horovod from Uber. >> https://horovod.readthedocs.io/en/stable/spark_include.html >> >> On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan wrote: >> > >> > Dear Spark User >> > >> > I am trying to parallelize the CNN (convolutional neural network) model >> using spark. I have developed the model using python and Keras library. The >> model works fine on a single machine but when we try on multiple machines, >> the execution time remains the same as sequential. >> > Could you please tell me that there is any built-in library for CNN to >> parallelize in spark framework. Moreover, MLLIB does not have any support >> for CNN. >> > Best regards >> > Mukhtaj >> > >> > >> > >> > >> >> - >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> -- *Patrick McCarthy * Senior Data Scientist, Machine Learning Engineering Dstillery 470 Park Ave South, 17th Floor, NYC 10016
Re: Issue in parallelization of CNN model using spark
link to a free book which may be useful. Hands-On Machine Learning with Scikit-Learn, Keras, and Tensorflow Concepts, Tools, and Techniques to Build Intelligent Systems by Aurélien Géron https://bit.ly/2zxueGt 13 Jul 2020, 15:18 Sean Owen, wrote: > There is a multilayer perceptron implementation in Spark ML, but > that's not what you're looking for. > To parallelize model training developed using standard libraries like > Keras, use Horovod from Uber. > https://horovod.readthedocs.io/en/stable/spark_include.html > > On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan wrote: > > > > Dear Spark User > > > > I am trying to parallelize the CNN (convolutional neural network) model > using spark. I have developed the model using python and Keras library. The > model works fine on a single machine but when we try on multiple machines, > the execution time remains the same as sequential. > > Could you please tell me that there is any built-in library for CNN to > parallelize in spark framework. Moreover, MLLIB does not have any support > for CNN. > > Best regards > > Mukhtaj > > > > > > > > > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: Issue in parallelization of CNN model using spark
This is very useful for me leading on from week4 of the Andrew Ng course. On Mon, 13 Jul 2020, 15:18 Sean Owen, wrote: > There is a multilayer perceptron implementation in Spark ML, but > that's not what you're looking for. > To parallelize model training developed using standard libraries like > Keras, use Horovod from Uber. > https://horovod.readthedocs.io/en/stable/spark_include.html > > On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan wrote: > > > > Dear Spark User > > > > I am trying to parallelize the CNN (convolutional neural network) model > using spark. I have developed the model using python and Keras library. The > model works fine on a single machine but when we try on multiple machines, > the execution time remains the same as sequential. > > Could you please tell me that there is any built-in library for CNN to > parallelize in spark framework. Moreover, MLLIB does not have any support > for CNN. > > Best regards > > Mukhtaj > > > > > > > > > > - > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
Re: Issue in parallelization of CNN model using spark
There is a multilayer perceptron implementation in Spark ML, but that's not what you're looking for. To parallelize model training developed using standard libraries like Keras, use Horovod from Uber. https://horovod.readthedocs.io/en/stable/spark_include.html On Mon, Jul 13, 2020 at 6:59 AM Mukhtaj Khan wrote: > > Dear Spark User > > I am trying to parallelize the CNN (convolutional neural network) model using > spark. I have developed the model using python and Keras library. The model > works fine on a single machine but when we try on multiple machines, the > execution time remains the same as sequential. > Could you please tell me that there is any built-in library for CNN to > parallelize in spark framework. Moreover, MLLIB does not have any support for > CNN. > Best regards > Mukhtaj > > > > - To unsubscribe e-mail: user-unsubscr...@spark.apache.org
Re: Issue in parallelization of CNN model using spark
Hi Mukhtaj, Parallelization on Spark is abstracted on the DataFrame. You can run anything locally on the driver but to make it run in parallel on the cluster you'll need to use the DataFrame abstraction. You may want to check maxpumperla/elephas. | | | | | | | | | | | maxpumperla/elephas Distributed Deep learning with Keras & Spark. Contribute to maxpumperla/elephas development by creating an accou... | | | Regards,Juan Martín. El lunes, 13 de julio de 2020 08:59:35 ART, Mukhtaj Khan escribió: Dear Spark User I am trying to parallelize the CNN (convolutional neural network) model using spark. I have developed the model using python and Keras library. The model works fine on a single machine but when we try on multiple machines, the execution time remains the same as sequential.Could you please tell me that there is any built-in library for CNN to parallelize in spark framework. Moreover, MLLIB does not have any support for CNN.Best regardsMukhtaj
