This is an automated email from the ASF dual-hosted git repository.
niketanpansare pushed a commit to branch gh-pages
in repository https://gitbox.apache.org/repos/asf/systemml.git
commit 47ce2179768b9dd2821aae7fc11f11c1fb8de210
Author: Niketan Pansare <npan...@us.ibm.com>
AuthorDate: Wed Mar 27 08:57:02 2019 -0700
[SYSTEMML-540] Updated Keras2DML to match Keras API and improved rmvar
performance
- Improved performance of rmvar by refactoring LocalVariableMap. With this
change, the end-to-end performance of a sample run of ResNet-200 improved from
55 seconds to 31 seconds.
- The parameters batch_size, max_iter, test_iter, test_interval, display of
Keras2DML constructor is removed. Instead, batch_size, epochs,
validation_split, validation_data parameters of fit() method.
- Updated the Caffe2DML generator to include the above parameters.
- Updated the documentation.
Closes #859.
---
beginners-guide-keras2dml.md | 165 ++++-----------------
gpu.md | 12 +-
index.md | 4 +-
reference-guide-caffe2dml.md | 68 +++++++++
...de-keras2dml.md => reference-guide-keras2dml.md | 142 +++---------------
5 files changed, 133 insertions(+), 258 deletions(-)
diff --git a/beginners-guide-keras2dml.md b/beginners-guide-keras2dml.md
index 2259397..788a489 100644
--- a/beginners-guide-keras2dml.md
+++ b/beginners-guide-keras2dml.md
@@ -27,34 +27,23 @@ limitations under the License.
<br/>
-## Introduction
+# Introduction
-Keras2DML is an **experimental API** that converts a Keras specification to
DML through the intermediate Caffe2DML module.
+Keras2DML converts a Keras specification to DML through the intermediate
Caffe2DML module.
It is designed to fit well into the mllearn framework and hence supports
NumPy, Pandas as well as PySpark DataFrame.
-### Getting Started
-
-To create a Keras2DML object, one needs to create a Keras model through the
Funcitonal API. please see the [Functional API.](https://keras.io/models/model/)
-This module utilizes the existing [Caffe2DML](beginners-guide-caffe2dml)
backend to convert Keras models into DML. Keras models are
-parsed and translated into Caffe prototext and caffemodel files which are then
piped into Caffe2DML. Thus one can follow the Caffe2DML
-documentation for further information.
-
-### Model Conversion
-
-Keras models are parsed based on their layer structure and corresponding
weights and translated into the relative Caffe layer and weight
-configuration. Be aware that currently this is a translation into Caffe and
there will be loss of information from keras models such as
-intializer information, and other layers which do not exist in Caffe.
-
First, install SystemML and other dependencies for the below demo:
```
-pip install systemml keras tensorflow mlxtend
+pip install systemml keras tensorflow
```
To create a Keras2DML object, simply pass the keras object to the Keras2DML
constructor. It's also important to note that your models
should be compiled so that the loss can be accessed for Caffe2DML.
+# Training Lenet on the MNIST dataset
+Download the MNIST dataset using [mlxtend
package](https://pypi.python.org/pypi/mlxtend).
```python
# pyspark --driver-memory 20g
@@ -115,130 +104,34 @@ sysml_model.fit(X_train, y_train)
sysml_model.score(X_test, y_test)
```
-# Frequently asked questions
-
-#### How can I get the training and prediction DML script for the Keras model?
-
-The training and prediction DML scripts can be generated using
`get_training_script()` and `get_prediction_script()` methods.
+# Prediction using a pretrained ResNet-50
```python
-from systemml.mllearn import Keras2DML
-sysml_model = Keras2DML(spark, keras_model, input_shape=(3,224,224))
-print(sysml_model.get_training_script())
-```
-
-#### What is the mapping between Keras' parameters and Caffe's solver
specification ?
-
-| | Specified via the
given parameter in the Keras2DML constructor | From input Keras' model
| Corresponding
parameter in the Caffe solver file |
-|--------------------------------------------------------|----------------------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------|
-| Solver type |
| `type(keras_model.optimizer)`.
Supported types: `keras.optimizers.{SGD, Adagrad, Adam}` | `type`
|
-| Maximum number of iterations | `max_iter`
| The `epoch` parameter in the `fit`
method is not supported. | `max_iter`
|
-| Validation dataset | `test_iter`
(explained in the below section) | The `validation_data`
parameter in the `fit` method is not supported. | `test_iter`
|
-| Monitoring the loss | `display,
test_interval` (explained in the below section) | The `LossHistory`
callback in the `fit` method is not supported. |
`display, test_interval` |
-| Learning rate schedule | `lr_policy`
| The `LearningRateScheduler`
callback in the `fit` method is not supported. | `lr_policy`
(default: step) |
-| Base learning rate |
| `keras_model.optimizer.lr`
| `base_lr`
|
-| Learning rate decay over each update |
| `keras_model.optimizer.decay`
| `gamma`
|
-| Global regularizer to use for all layers |
`regularization_type,weight_decay` | The current
version of Keras2DML doesnot support custom regularizers per layer. |
`regularization_type,weight_decay` |
-| If type of the optimizer is `keras.optimizers.SGD` |
| `momentum, nesterov`
| `momentum, type`
|
-| If type of the optimizer is `keras.optimizers.Adam` |
| `beta_1, beta_2, epsilon`. The
parameter `amsgrad` is not supported. | `momentum,
momentum2, delta` |
-| If type of the optimizer is `keras.optimizers.Adagrad` |
| `epsilon`
| `delta`
|
-
-#### How do I specify the batch size and the number of epochs ?
+# pyspark --driver-memory 20g
+# Disable Tensorflow from using GPU to avoid unnecessary evictions by SystemML
runtime
+import os
+os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
+os.environ['CUDA_VISIBLE_DEVICES'] = ''
-Since Keras2DML is a mllearn API, it doesnot accept the batch size and number
of epochs as the parameter in the `fit` method.
-Instead, these parameters are passed via `batch_size` and `max_iter`
parameters in the Keras2DML constructor.
-For example, the equivalent Python code for `keras_model.fit(features, labels,
epochs=10, batch_size=64)` is as follows:
+# Set channel first layer
+from keras import backend as K
+K.set_image_data_format('channels_first')
-```python
from systemml.mllearn import Keras2DML
-epochs = 10
-batch_size = 64
-num_samples = features.shape[0]
-max_iter = int(epochs*math.ceil(num_samples/batch_size))
-sysml_model = Keras2DML(spark, keras_model, batch_size=batch_size,
max_iter=max_iter, ...)
-sysml_model.fit(features, labels)
-```
-
-#### What optimizer and loss does Keras2DML use by default if `keras_model` is
not compiled ?
-
-If the user does not `compile` the keras model, then we throw an error.
-
-For classification applications, you can consider using cross entropy loss and
SGD optimizer with nesterov momentum:
-
-```python
-keras_model.compile(loss='categorical_crossentropy',
optimizer=keras.optimizers.SGD(lr=0.01, momentum=0.95, decay=5e-4,
nesterov=True))
+import systemml as sml
+import keras, urllib
+from PIL import Image
+from keras.applications.resnet50 import preprocess_input, decode_predictions,
ResNet50
+
+keras_model =
ResNet50(weights='imagenet',include_top=True,pooling='None',input_shape=(3,224,224))
+keras_model.compile(optimizer='sgd', loss= 'categorical_crossentropy')
+
+sysml_model = Keras2DML(spark,keras_model,input_shape=(3,224,224),
weights='weights_dir',
labels='https://raw.githubusercontent.com/apache/systemml/master/scripts/nn/examples/caffe2dml/models/imagenet/labels.txt')
+sysml_model.summary()
+urllib.urlretrieve('https://upload.wikimedia.org/wikipedia/commons/f/f4/Cougar_sitting.jpg',
'test.jpg')
+img_shape = (3, 224, 224)
+input_image = sml.convertImageToNumPyArr(Image.open('test.jpg'),
img_shape=img_shape)
+sysml_model.predict(input_image)
```
-Please refer to [Keras's documentation](https://keras.io/losses/) for more
detail.
-
-#### What is the learning rate schedule used ?
-
-Keras2DML does not support the `LearningRateScheduler` callback.
-Instead one can set the custom learning rate schedule to one of the following
schedules by using the `lr_policy` parameter of the constructor:
-- `step`: return `base_lr * gamma ^ (floor(iter / step))` (default schedule)
-- `fixed`: always return `base_lr`.
-- `exp`: return `base_lr * gamma ^ iter`
-- `inv`: return `base_lr * (1 + gamma * iter) ^ (- power)`
-- `poly`: the effective learning rate follows a polynomial decay, to be zero
by the max_iter. return `base_lr (1 - iter/max_iter) ^ (power)`
-- `sigmoid`: the effective learning rate follows a sigmod decay return
b`ase_lr ( 1/(1 + exp(-gamma * (iter - stepsize))))`
-
-#### How to set the size of the validation dataset ?
-
-The size of the validation dataset is determined by the parameters `test_iter`
and the batch size. For example: If the batch size is 64 and
-`test_iter` is set to 10 in the `Keras2DML`'s constructor, then the validation
size is 640. This setting generates following DML code internally:
-
-```python
-num_images = nrow(y_full)
-BATCH_SIZE = 64
-num_validation = 10 * BATCH_SIZE
-X = X_full[(num_validation+1):num_images,]; y =
y_full[(num_validation+1):num_images,]
-X_val = X_full[1:num_validation,]; y_val = y_full[1:num_validation,]
-num_images = nrow(y)
-```
-
-#### How to monitor loss via command-line ?
-
-To monitor loss, please set the parameters `display`, `test_iter` and
`test_interval` in the `Keras2DML`'s constructor.
-For example: for the expression `Keras2DML(..., display=100, test_iter=10,
test_interval=500)`, we
-- display the training loss and accuracy every 100 iterations and
-- carry out validation every 500 training iterations and display validation
loss and accuracy.
-
-#### How do you ensure that Keras2DML produce same results as other Keras'
backend?
-
-To verify that Keras2DML produce same results as other Keras' backend, we have
[Python unit
tests](https://github.com/apache/systemml/blob/master/src/main/python/tests/test_nn_numpy.py)
-that compare the results of Keras2DML with that of TensorFlow. We assume that
Keras team ensure that all their backends are consistent with their TensorFlow
backend.
-
-#### How can I train very deep models on GPU?
-
-Unlike Keras where default train and test algorithm is `minibatch`, you can
specify the
-algorithm using the parameters `train_algo` and `test_algo` (valid values are:
`minibatch`, `allreduce_parallel_batches`,
-`looped_minibatch`, and `allreduce`). Here are some common settings:
-
-| |
PySpark script
| Changes to
Network/Solver |
-|--------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------|
-| Single-node CPU execution (similar to Caffe with solver_mode: CPU) |
`lenet.set(train_algo="minibatch", test_algo="minibatch")`
| Ensure that
`batch_size` is set to appropriate value (for example: 64) |
-| Single-node single-GPU execution |
`lenet.set(train_algo="minibatch",
test_algo="minibatch").setGPU(True).setForceGPU(True)`
| Ensure that `batch_size` is set to appropriate value
(for example: 64) |
-| Single-node multi-GPU execution (similar to Caffe with solver_mode: GPU) |
`lenet.set(train_algo="allreduce_parallel_batches", test_algo="minibatch",
parallel_batches=num_gpu).setGPU(True).setForceGPU(True)` | Ensure that
`batch_size` is set to appropriate value (for example: 64) |
-| Distributed prediction |
`lenet.set(test_algo="allreduce")`
|
|
-| Distributed synchronous training |
`lenet.set(train_algo="allreduce_parallel_batches",
parallel_batches=num_cluster_cores)`
| Ensure that `batch_size` is set to appropriate value (for example: 64) |
-
-Here are high-level guidelines to train very deep models on GPU with Keras2DML
(and Caffe2DML):
-
-1. If there exists at least one layer/operator that does not fit on the
device, please allow SystemML's optimizer to perform operator placement based
on the memory estimates `sysml_model.setGPU(True)`.
-2. If each individual layer/operator fits on the device but not the entire
network with a batch size of 1, then
-- Rely on SystemML's GPU Memory Manager to perform automatic eviction
(recommended): `sysml_model.setGPU(True) # Optional: .setForceGPU(True)`
-- Or enable Nvidia's Unified Memory:
`sysml_model.setConfigProperty('sysml.gpu.memory.allocator', 'unified_memory')`
-3. If the entire neural network does not fit in the GPU memory with the
user-specified `batch_size`, but fits in the GPU memory with `local_batch_size`
such that `1 << local_batch_size < batch_size`, then
-- Use either of the above two options.
-- Or enable `train_algo` that performs multiple forward-backward pass with
batch size `local_batch_size`, aggregate gradients and finally updates the
model:
-```python
-sysml_model = Keras2DML(spark, keras_model, batch_size=local_batch_size)
-sysml_model.set(train_algo="looped_minibatch",
parallel_batches=int(batch_size/local_batch_size))
-sysml_model.setGPU(True).setForceGPU(True)
-```
-- Or add `int(batch_size/local_batch_size)` GPUs and perform single-node
multi-GPU training with batch size `local_batch_size`:
-```python
-sysml_model = Keras2DML(spark, keras_model, batch_size=local_batch_size)
-sysml_model.set(train_algo="allreduce_parallel_batches",
parallel_batches=int(batch_size/local_batch_size))
-sysml_model.setGPU(True).setForceGPU(True)
-```
+Please see [Keras2DML's reference
guide](http://apache.github.io/systemml/reference-guide-keras2dml) for more
details.
diff --git a/gpu.md b/gpu.md
index f334b47..c5cdb56 100644
--- a/gpu.md
+++ b/gpu.md
@@ -185,6 +185,16 @@ $ ./bin/x86_64/linux/release/deviceQuery
$ ./bin/x86_64/linux/release/bandwidthTestÂ
$ ./bin/x86_64/linux/release/matrixMulCUBLASÂ
```
+- Test CUDA and CuDNN with SystemML
+```
+$ git clone https://github.com/apache/systemml.git
+$ cd systemml
+$ mvn -Dit.test=org.apache.sysml.test.gpu.AggregateTernaryTests verify
-PgpuTests
+$ mvn -Dit.test=org.apache.sysml.test.gpu.NeuralNetworkOpTests verify
-PgpuTests
+```
+
+If you get an `java.lang.UnsatisfiedLinkError: libcusparse.so.9.0: cannot open
shared object file: No such file or directory` error, then
+CUDA toolkit is not installed correctly or it is not included in the
`LD_LIBRARY_PATH`.
### How to install CUDA 9 on Centos 7 with yum?
@@ -211,4 +221,4 @@ cd gcc-5.3.0
num_cores=`grep -c ^processor /proc/cpuinfo`
make -j $num_cores
sudo make install
-```
+```
\ No newline at end of file
diff --git a/index.md b/index.md
index 4ceaee6..3169b15 100644
--- a/index.md
+++ b/index.md
@@ -76,8 +76,8 @@ for running SystemML from Spark via Scala, Python, or Java.
machine in R-like and Python-like declarative languages.
* [JMLC](jmlc) - Java Machine Learning Connector.
* [Deep Learning with SystemML](deep-learning)
- * *Experimental* Caffe2DML API for Deep Learning ([beginner's
guide](beginners-guide-caffe2dml), [reference
guide](reference-guide-caffe2dml)) - Converts a Caffe specification to DML.
- * *Experimental* [Keras2DML API](beginners-guide-keras2dml) for Deep
Learning.
+ * Keras2DML API for Deep Learning ([beginner's
guide](beginners-guide-keras2dml), [reference
guide](reference-guide-keras2dml)) - Converts a Keras model to DML.
+ * Caffe2DML API for Deep Learning ([beginner's
guide](beginners-guide-caffe2dml), [reference
guide](reference-guide-caffe2dml)) - Converts a Caffe specification to DML.
## Language Guides
diff --git a/reference-guide-caffe2dml.md b/reference-guide-caffe2dml.md
index 6242e03..1a3d154 100644
--- a/reference-guide-caffe2dml.md
+++ b/reference-guide-caffe2dml.md
@@ -1135,3 +1135,71 @@ class precision recall f1-score
num_true_labels
...
```
+#### Design document of Caffe2DML
+
+1. Caffe2DML is designed to fit well into the mllearn framework. Hence, the
key methods that were to be implemented are:
+- `getTrainingScript` for the `Estimator` class.
+- `getPredictionScript` for the `Model` class.
+
+These methods should be the starting point of any developer to understand the
DML generated for training and prediction respectively.
+
+2. To simplify the DML generation in `getTrainingScript` and
`getPredictionScript method`, we use DMLGenerator interface.
+This interface generates DML string for common operations such as loops (such
as if, for, while) as well as built-in functions (read, write), etc.
+Also, this interface helps in "code reading" of the Caffe2DML class.
+
+3. Here is an analogy for SystemML developers to think of various moving
components of Caffe2DML:
+- Like `Dml.g4` in the `org.apache.sysml.parser.dml` package, `caffe.proto` in
the `src/main/proto/caffe` directory
+is used to generate classes to parse the input files.
+
+```
+Dml.g4 ---> antlr ---> DmlLexer.java, DmlListener.java, DmlParser.java
+caffe.proto ---> protoc ---> target/generated-sources/caffe/Caffe.java
+```
+
+- Just like the classes generated by Dml.g4 are used to parse input DML file,
+the `target/generated-sources/caffe/Caffe.java` class is used to parse the
input caffe network/deploy prototxt and solver files.
+
+- You can think of `.caffemodel` file as DML file with matrix values encoded
in it (please see below example).
+So it is possible to read `.caffemodel` file with the `Caffe.java` class. This
is done in Utils.scala's `readCaffeNet` method.
+
+```
+X = matrix("1.2 3.5 0.999 7.123", rows=2, cols=2)
+...
+```
+
+- Just like we convert the AST generated by antlr into our DMLProgram
representation, we convert
+caffe's abstraction into the below given mapping classes for layer, solver and
learning rate.
+These mapping classes maps the corresponding Caffe abstraction to the
SystemML-NN library.
+This greatly simplifies adding new layers into Caffe2DML:
+```
+trait CaffeLayer {
+ // Any layer that wants to reuse SystemML-NN has to override following
methods that help in generating the DML for the given layer:
+ def sourceFileName:String;
+ def init(dmlScript:StringBuilder):Unit;
+ def forward(dmlScript:StringBuilder, isPrediction:Boolean):Unit;
+ def backward(dmlScript:StringBuilder, outSuffix:String):Unit;
+ ...
+}
+trait CaffeSolver {
+ def sourceFileName:String;
+ def update(dmlScript:StringBuilder, layer:CaffeLayer):Unit;
+ def init(dmlScript:StringBuilder, layer:CaffeLayer):Unit;
+}
+```
+
+4. To simplify the traversal of the network, we created a Network interface:
+```
+trait Network {
+ def getLayers(): List[String]
+ def getCaffeLayer(layerName:String):CaffeLayer
+ def getBottomLayers(layerName:String): Set[String]
+ def getTopLayers(layerName:String): Set[String]
+ def getLayerID(layerName:String): Int
+}
+```
+
+5. One of the key design restriction of Caffe2DML is that every layer is
identified uniquely by its name.
+This restriction simplifies the code significantly.
+To shield from network files that violates this restriction, Caffe2DML
performs rewrites in CaffeNetwork class (search for condition 1-5 in Caffe2DML
class).
+
+6. Like Caffe, Caffe2DML also expects the layers to be in sorted order.
diff --git a/beginners-guide-keras2dml.md b/reference-guide-keras2dml.md
similarity index 67%
copy from beginners-guide-keras2dml.md
copy to reference-guide-keras2dml.md
index 2259397..a576ee7 100644
--- a/beginners-guide-keras2dml.md
+++ b/reference-guide-keras2dml.md
@@ -27,93 +27,10 @@ limitations under the License.
<br/>
-## Introduction
-Keras2DML is an **experimental API** that converts a Keras specification to
DML through the intermediate Caffe2DML module.
-It is designed to fit well into the mllearn framework and hence supports
NumPy, Pandas as well as PySpark DataFrame.
+# Layers supported in Keras2DML
-### Getting Started
-
-To create a Keras2DML object, one needs to create a Keras model through the
Funcitonal API. please see the [Functional API.](https://keras.io/models/model/)
-This module utilizes the existing [Caffe2DML](beginners-guide-caffe2dml)
backend to convert Keras models into DML. Keras models are
-parsed and translated into Caffe prototext and caffemodel files which are then
piped into Caffe2DML. Thus one can follow the Caffe2DML
-documentation for further information.
-
-### Model Conversion
-
-Keras models are parsed based on their layer structure and corresponding
weights and translated into the relative Caffe layer and weight
-configuration. Be aware that currently this is a translation into Caffe and
there will be loss of information from keras models such as
-intializer information, and other layers which do not exist in Caffe.
-
-First, install SystemML and other dependencies for the below demo:
-
-```
-pip install systemml keras tensorflow mlxtend
-```
-
-To create a Keras2DML object, simply pass the keras object to the Keras2DML
constructor. It's also important to note that your models
-should be compiled so that the loss can be accessed for Caffe2DML.
-
-
-
-```python
-# pyspark --driver-memory 20g
-
-# Disable Tensorflow from using GPU to avoid unnecessary evictions by SystemML
runtime
-import os
-os.environ['CUDA_DEVICE_ORDER'] = 'PCI_BUS_ID'
-os.environ['CUDA_VISIBLE_DEVICES'] = ''
-
-# Import dependencies
-from mlxtend.data import mnist_data
-import numpy as np
-from sklearn.utils import shuffle
-from keras.models import Sequential
-from keras.layers import Input, Dense, Conv2D, MaxPooling2D, Dropout,Flatten
-from keras import backend as K
-from keras.models import Model
-from keras.optimizers import SGD
-
-# Set channel first layer
-K.set_image_data_format('channels_first')
-
-# Download the MNIST dataset
-X, y = mnist_data()
-X, y = shuffle(X, y)
-
-# Split the data into training and test
-n_samples = len(X)
-X_train = X[:int(.9 * n_samples)]
-y_train = y[:int(.9 * n_samples)]
-X_test = X[int(.9 * n_samples):]
-y_test = y[int(.9 * n_samples):]
-
-# Define Lenet in Keras
-keras_model = Sequential()
-keras_model.add(Conv2D(32, kernel_size=(5, 5), activation='relu',
input_shape=(1,28,28), padding='same'))
-keras_model.add(MaxPooling2D(pool_size=(2, 2)))
-keras_model.add(Conv2D(64, (5, 5), activation='relu', padding='same'))
-keras_model.add(MaxPooling2D(pool_size=(2, 2)))
-keras_model.add(Flatten())
-keras_model.add(Dense(512, activation='relu'))
-keras_model.add(Dropout(0.5))
-keras_model.add(Dense(10, activation='softmax'))
-keras_model.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01,
decay=1e-6, momentum=0.9, nesterov=True))
-keras_model.summary()
-
-# Scale the input features
-scale = 0.00390625
-X_train = X_train*scale
-X_test = X_test*scale
-
-# Train Lenet using SystemML
-from systemml.mllearn import Keras2DML
-sysml_model = Keras2DML(spark, keras_model, weights='weights_dir')
-# sysml_model.setConfigProperty("sysml.native.blas", "auto")
-# sysml_model.setGPU(True).setForceGPU(True)
-sysml_model.fit(X_train, y_train)
-sysml_model.score(X_test, y_test)
-```
+TODO:
# Frequently asked questions
@@ -132,7 +49,6 @@ print(sysml_model.get_training_script())
| | Specified via the
given parameter in the Keras2DML constructor | From input Keras' model
| Corresponding
parameter in the Caffe solver file |
|--------------------------------------------------------|----------------------------------------------------------------|-----------------------------------------------------------------------------------------|--------------------------------------------------|
| Solver type |
| `type(keras_model.optimizer)`.
Supported types: `keras.optimizers.{SGD, Adagrad, Adam}` | `type`
|
-| Maximum number of iterations | `max_iter`
| The `epoch` parameter in the `fit`
method is not supported. | `max_iter`
|
| Validation dataset | `test_iter`
(explained in the below section) | The `validation_data`
parameter in the `fit` method is not supported. | `test_iter`
|
| Monitoring the loss | `display,
test_interval` (explained in the below section) | The `LossHistory`
callback in the `fit` method is not supported. |
`display, test_interval` |
| Learning rate schedule | `lr_policy`
| The `LearningRateScheduler`
callback in the `fit` method is not supported. | `lr_policy`
(default: step) |
@@ -143,21 +59,11 @@ print(sysml_model.get_training_script())
| If type of the optimizer is `keras.optimizers.Adam` |
| `beta_1, beta_2, epsilon`. The
parameter `amsgrad` is not supported. | `momentum,
momentum2, delta` |
| If type of the optimizer is `keras.optimizers.Adagrad` |
| `epsilon`
| `delta`
|
-#### How do I specify the batch size and the number of epochs ?
+#### How do I specify the batch size and the number of epochs?
-Since Keras2DML is a mllearn API, it doesnot accept the batch size and number
of epochs as the parameter in the `fit` method.
-Instead, these parameters are passed via `batch_size` and `max_iter`
parameters in the Keras2DML constructor.
-For example, the equivalent Python code for `keras_model.fit(features, labels,
epochs=10, batch_size=64)` is as follows:
+Like Keras, the user can provide `batch_size` and `epochs` via the `fit`
method. For example: `sysml_model.fit(features, labels, epochs=10,
batch_size=64)`.
-```python
-from systemml.mllearn import Keras2DML
-epochs = 10
-batch_size = 64
-num_samples = features.shape[0]
-max_iter = int(epochs*math.ceil(num_samples/batch_size))
-sysml_model = Keras2DML(spark, keras_model, batch_size=batch_size,
max_iter=max_iter, ...)
-sysml_model.fit(features, labels)
-```
+Note, we do not support `verbose` and `callbacks` parameters in our `fit`
method. Please use SparkContext's `setLogLevel` method to control the verbosity.
#### What optimizer and loss does Keras2DML use by default if `keras_model` is
not compiled ?
@@ -184,24 +90,9 @@ Instead one can set the custom learning rate schedule to
one of the following sc
#### How to set the size of the validation dataset ?
-The size of the validation dataset is determined by the parameters `test_iter`
and the batch size. For example: If the batch size is 64 and
-`test_iter` is set to 10 in the `Keras2DML`'s constructor, then the validation
size is 640. This setting generates following DML code internally:
-
-```python
-num_images = nrow(y_full)
-BATCH_SIZE = 64
-num_validation = 10 * BATCH_SIZE
-X = X_full[(num_validation+1):num_images,]; y =
y_full[(num_validation+1):num_images,]
-X_val = X_full[1:num_validation,]; y_val = y_full[1:num_validation,]
-num_images = nrow(y)
-```
-
-#### How to monitor loss via command-line ?
-
-To monitor loss, please set the parameters `display`, `test_iter` and
`test_interval` in the `Keras2DML`'s constructor.
-For example: for the expression `Keras2DML(..., display=100, test_iter=10,
test_interval=500)`, we
-- display the training loss and accuracy every 100 iterations and
-- carry out validation every 500 training iterations and display validation
loss and accuracy.
+Like Keras, the validation dataset can be set in two ways:
+1. `validation_split` parameter (of type `float` between 0 and 1) in the `fit`
method: It is the fraction of the training data to be used as validation data.
The model will set apart this fraction of the training data, will not train on
it, and will evaluate the loss and any model metrics on this data at the end of
each epoch.
+2. `validation_data` parameter (of type `(x_val, y_val)` where `x_val` and
`y_val` are NumPy arrays) in the `fit` method: on which to evaluate the loss at
the end of each epoch. The model will not be trained on this data.
validation_data will override validation_split.
#### How do you ensure that Keras2DML produce same results as other Keras'
backend?
@@ -232,13 +123,26 @@ Here are high-level guidelines to train very deep models
on GPU with Keras2DML (
- Use either of the above two options.
- Or enable `train_algo` that performs multiple forward-backward pass with
batch size `local_batch_size`, aggregate gradients and finally updates the
model:
```python
-sysml_model = Keras2DML(spark, keras_model, batch_size=local_batch_size)
+sysml_model = Keras2DML(spark, keras_model)
sysml_model.set(train_algo="looped_minibatch",
parallel_batches=int(batch_size/local_batch_size))
sysml_model.setGPU(True).setForceGPU(True)
+sysml_model.fit(X, y, batch_size=local_batch_size)
```
- Or add `int(batch_size/local_batch_size)` GPUs and perform single-node
multi-GPU training with batch size `local_batch_size`:
```python
-sysml_model = Keras2DML(spark, keras_model, batch_size=local_batch_size)
+sysml_model = Keras2DML(spark, keras_model)
sysml_model.set(train_algo="allreduce_parallel_batches",
parallel_batches=int(batch_size/local_batch_size))
sysml_model.setGPU(True).setForceGPU(True)
+sysml_model.fit(X, y, batch_size=local_batch_size)
```
+
+#### Design document of Keras2DML.
+
+Keras2DML internally utilizes the existing
[Caffe2DML](beginners-guide-caffe2dml) backend to convert Keras models into
DML. Keras models are
+parsed and translated into Caffe prototext and caffemodel files which are then
piped into Caffe2DML.
+
+Keras models are parsed based on their layer structure and corresponding
weights and translated into the relative Caffe layer and weight
+configuration. Be aware that currently this is a translation into Caffe and
there will be loss of information from keras models such as
+intializer information, and other layers which do not exist in Caffe.
+
+Read the [Caffe2DML's reference
guide](http://apache.github.io/systemml/reference-guide-caffe2dml) for the
design documentation.
\ No newline at end of file
