[
https://issues.apache.org/jira/browse/MADLIB-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16791955#comment-16791955
]
Frank McQuillan commented on MADLIB-1268:
-----------------------------------------
!training time vs accuracy for different cluster sizes.png!
> Spike - CNN convergence, data parallel with merge
> -------------------------------------------------
>
> Key: MADLIB-1268
> URL: https://issues.apache.org/jira/browse/MADLIB-1268
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Deep Learning
> Reporter: Frank McQuillan
> Assignee: Frank McQuillan
> Priority: Major
> Fix For: v1.16
>
> Attachments: CIFAR-10 validation accuracy by iteration.png,
> screenshot-1.png, screenshot-2.png, training time vs accuracy for different
> cluster sizes.png
>
>
> Story
> `As a MADlib developer`
> I want investigate convergence behavior when running a single distributed CNN
> model across the Greenplum cluster using Keras with a Tensorflow backend
> `so that`
> I can see if it converges in a predictable and expected way.
> Details
> * By "single distributed CNN model" I mean data parallel with merge (not
> model parallel) [6,7,8].
> * In defining the merge function, review [1] for single-server, multi-GPU
> merge function, or use standard MADlib weighted average approach.
> * For dataset, consider MNIST and/or CIFAR-10. A bigger data set like Places
> http://places2.csail.mit.edu/ may also be useful.
> Acceptance
> 1) Plot characteristic curves of loss vs. iteration number. Compare with
> MADlib merge (this story) vs. without merge.
> 2) Define what the merge function is for CNN. Is it the same as [1] or
> something else? Does it operate on weights only or does it need gradients?
> 3) What does the architecture look like? Draw a diagram showing sync/merge
> step for distributed model training.
> 4) What tests do we need to do to convince ourselves that the architecture is
> valid?
> 5) Do we need to write different merge functions, or have a different
> approach, for each different neural net type algorithm? Or is there a
> general approach that we can use that will apply to this class of algorithms?
> References
> [1] Check for “# Merge outputs under expected scope” section in the python
> program
>
> https://github.com/keras-team/keras/blob/bf1378f39d02b7d0b53ece5458f9275ac8208046/keras/utils/multi_gpu_utils.py
> [2] Single Machine Data Parallel multi GPU Training
> https://www.pyimagesearch.com/2017/10/30/how-to-multi-gpu-training-with-keras-python-and-deep-learning/
> [3] Why are GPUs necessary for training Deep Learning models?
> https://www.analyticsvidhya.com/blog/2017/05/gpus-necessary-for-deep-learning/
> [4] Deep Learning vs Classical Machine Learning
> https://towardsdatascience.com/deep-learning-vs-classical-machine-learning-9a42c6d48aa
> [5] TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed
> Systems
> https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf
> [6] Demystifying Parallel and Distributed Deep Learning: An In-Depth
> Concurrency Analysis
> https://arxiv.org/pdf/1802.09941.pdf
> * see section 7.4.2 for discussion on model averaging
> [7] Deep learning with Elastic Averaging SGD
> https://papers.nips.cc/paper/5761-deep-learning-with-elastic-averaging-sgd.pdf
> * use momentum and Nesterov methods in model averaging computation
> [8] SCALABLE TRAINING OF DEEP LEARNING MACHINES BY INCREMENTAL BLOCK
> TRAINING WITH INTRA-BLOCK PARALLEL OPTIMIZATION AND BLOCKWISE
> MODEL-UPDATE FILTERING
> https://www.microsoft.com/en-us/research/wp-content/uploads/2016/08/0005880.pdf
> * similar to [7], uses momentum methods
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
