Frank McQuillan created MADLIB-1268:
---------------------------------------
Summary: Spike - CNN convergence, data parallel with merge
Key: MADLIB-1268
URL: https://issues.apache.org/jira/browse/MADLIB-1268
Project: Apache MADlib
Issue Type: New Feature
Reporter: Frank McQuillan
Fix For: v2.0
Story
`As a MADlib developer`
I want investigate convergence behaviour when running a single distributed CNN
model across the Greenplum cluster using Keras with a Tensorflow backend
`so that`
I can see if it converges in a predictable and expected way.
Details
* By "single distributed CNN model" I mean data parallel with merge (not model
parallel).
* Does not need to use an aggregate for this spike, if that is too
inconvenient, since performance is not the focus of this story. It's about
convergence.
* In defining the merge function, review [2] for single-server, multi-GPU merge
function. Perhaps we can do the exact same thing for multi-server?
* For dataset, consider using Pavan's CNN code and data set [3]. Another
option is MNIST and/or CIFAR-10.
* See page 11 of [8] re synchronous data parallel in TF
Acceptance
1) Plot characteristic curves of loss vs. iteration number. Compare with
MADlib merge (this story) vs. without merge.
2) Define what the merge function is for CNN. Is it the same as [3] or
something else? Does it operate on weights only or does it need gradients?
3) What does the architecture look like? Draw a diagram showing sync/merge
step for distributed model training.
4) What tests do we need to do to convince ourselves that the architecture is
valid?
5) Do we need to write different merge functions, or have a different approach,
for each different neural net type algorithm? Or is there a general approach
that we can use that will apply to this class of algorithms?
6) Anything to learn from pg-strom [2] ?
7) Anything to learn from H20 [3] ? I don’t think they are doing distributed
training, rather grid search and such.
References
[2] Check for “# Merge outputs under expected scope” section in the python
program
https://github.com/keras-team/keras/blob/bf1378f39d02b7d0b53ece5458f9275ac8208046/keras/utils/multi_gpu_utils.py
[3] Deep learning example for image segmentation
https://drive.google.com/drive/folders/1mgZPGuDP1JI1TUVaRndexDZTlSABLci9?usp=sharing
[4] Deep Learning & Greenplum Discussion (Pavan/Pivotal DE)
https://drive.google.com/file/d/1U808PAwMetNL38mrboPHdn8RKOrpQyBz/view?usp=sharing
[5] Single Machine Data Parallel multi GPU Training
https://www.pyimagesearch.com/2017/10/30/how-to-multi-gpu-training-with-keras-python-and-deep-learning/
[6] Why are GPUs necessary for training Deep Learning models?
https://www.analyticsvidhya.com/blog/2017/05/gpus-necessary-for-deep-learning/
[7] Deep Learning vs Classical Machine Learning
https://towardsdatascience.com/deep-learning-vs-classical-machine-learning-9a42c6d48aa
[8] TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed
Systems
https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/45166.pdf
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
