[
https://issues.apache.org/jira/browse/MADLIB-1338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851381#comment-16851381
]
Frank McQuillan edited comment on MADLIB-1338 at 5/29/19 11:14 PM:
-------------------------------------------------------------------
LGTM
Re-tested, all look OK except (4) which can be cleaned up in a later story.
(1)
What is the final list of metrics that are supported and those that are not
supported?
See:
https://www.pivotaltracker.com/n/projects/695473/stories/165776969/comments/202255072
https://docs.google.com/document/d/1RXC8bA4vZ3H3AmGbYtLHqi4fZ7WcMCVSXWNiSwRfZvo/edit
loss function not supported
'sparse_categorical_crossentropy'
metrics not supported
'sparse_categorical_accuracy'
'top_k_categorical_accuracy'
'sparse_top_k_categorical_accuracy'
(2)
fit with validation dataset, compute metrics at the end
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.54990196228 sec, Total was
0.758991956711 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.555626153946 sec, Total was
0.75802898407 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 2.43494582176 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.620418787003 sec, Total was
0.787483930588 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.660104036331 sec, Total was
0.832265853882 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 0.834777832031 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.574198007584 sec, Total was
0.753618955612 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.534913063049 sec, Total was
0.704334020615 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.756074905396 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.210503101349 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 0.324999988079.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 1.0396116972.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.190848112106 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 0.366666674614.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 1.06409192085.
CONTEXT: PL/Python function "madlib_keras_fit"
madlib_keras_fit
------------------
(1 row)
Time: 6817.188 ms
{code}
Verbose output: FIXED
INFO: Time for training in iteration 3: 0.756074905396 sec
INFO: Time for evaluation in iteration 3: 0.210503101349 sec.
Clarified training vs evaluation in INFO message: FIXED
(3)
fit with validation dataset, compute metrics at the end (con't)
{code}
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-23 22:33:08.611677
end_training_time | 2019-05-23 22:33:13.250892
metrics_elapsed_time | {4.63921880722046}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.324999988079
training_loss_final | 1.0396116972
training_metrics | {0.324999988079071}
training_loss | {1.03961169719696}
validation_metrics_final | 0.366666674614
validation_loss_final | 1.06409192085
validation_metrics | {0.366666674613953}
validation_loss | {1.06409192085266}
metrics_iters | {3}
{code}
Change to elapsed time looks better:
start_training_time | 2019-05-23 22:33:08.611677
end_training_time | 2019-05-23 22:33:13.250892
metrics_elapsed_time | {4.63921880722046}
13.250892 - 8.611677 = 4.639215
OK
(4)
fit with validation dataset, compute metrics every iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
1, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.535367012024 sec, Total was
0.746275901794 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.554301023483 sec, Total was
0.752547979355 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 3.40036702156 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.195158958435 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 1: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 1: 0.938332438469.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.332806825638 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 1: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 1: 0.944462954998.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.887582063675 sec, Total was
1.15123796463 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.860605955124 sec, Total was
1.12788796425 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 1.15389895439 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.192572116852 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 2: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 2: 0.900591552258.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.212724208832 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 2: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 2: 0.912341833115.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.607415914536 sec, Total was
0.850666046143 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.609027862549 sec, Total was
0.850383996964 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.852729082108 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.213857889175 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 0.865503013134.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.254362106323 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 0.878823339939.
CONTEXT: PL/Python function "madlib_keras_fit"
-[ RECORD 1 ]----+-
madlib_keras_fit |
Time: 7409.327 ms
{code}
What is this saying?
INFO: Processed 60 images: Fit took 0.535367012024 sec, Total was
0.746275901794 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.554301023483 sec, Total was
0.752547979355 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
Why does 'time for evaluation' repeat for every iteration and have different
times? e.g.,
INFO: Time for evaluation in iteration 3: 0.213857889175 sec.
INFO: Time for evaluation in iteration 3: 0.254362106323 sec.
{code}
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 1
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-23 22:38:33.342727
end_training_time | 2019-05-23 22:38:39.344795
metrics_elapsed_time | {3.45014500617981,4.83612895011902,6.00208592414856}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.358333319426
training_loss_final | 1.15425133705
training_metrics |
{0.358333319425583,0.358333319425583,0.358333319425583}
training_loss | {1.35349214076996,1.23959410190582,1.15425133705139}
validation_metrics_final | 0.233333334327
validation_loss_final | 1.31097722054
validation_metrics |
{0.233333334326744,0.233333334326744,0.233333334326744}
validation_loss | {1.57807862758636,1.42993664741516,1.31097722053528}
metrics_iters | {1,2,3}
{code}
Time intervals look OK:
metrics_elapsed_time = start_training_time - start_training_time
6.00208592414856 = 22:38:39.344795 - 22:38:33.342727
(5)
fit with validation dataset, compute metrics at different iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
2, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 2
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:01:42.08684
end_training_time | 2019-05-29 20:01:48.658938
metrics_elapsed_time | {5.42638921737671,6.57209706306458}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.341666668653
training_loss_final | 1.35804212093
training_metrics | {0.349999994039536,0.341666668653488}
training_loss | {1.42269325256348,1.35804212093353}
validation_metrics_final | 0.266666680574
validation_loss_final | 1.50677502155
validation_metrics | {0.266666680574417,0.266666680574417}
validation_loss | {1.58999609947205,1.50677502155304}
metrics_iters | {2,3}
Time: 22.104 ms
{code}
Looks OK {2,3}
(6)
fit with no validation dataset, compute metrics at different iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
NULL, -- validation_table
2, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table |
metrics_compute_frequency | 2
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:02:54.856595
end_training_time | 2019-05-29 20:03:01.289986
metrics_elapsed_time | {4.89847302436829,6.4333918094635}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.316666662693
training_loss_final | 2.19523310661
training_metrics | {0.316666662693024,0.316666662693024}
training_loss | {2.31425333023071,2.19523310661316}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
metrics_iters | {2,3}
{code}
Looks OK
(7)
a different metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['mae'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+---------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['mae']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:05:39.270399
end_training_time | 2019-05-29 20:05:42.27942
metrics_elapsed_time | {3.00901985168457}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {mae}
training_metrics_final | 0.46932977438
training_loss_final | 1.41450297832
training_metrics | {0.46932977437973}
training_loss | {1.41450297832489}
validation_metrics_final | 0.461133390665
validation_loss_final | 1.38406682014
validation_metrics | {0.461133390665054}
validation_loss | {1.38406682014465}
metrics_iters | {3}
{code}
Looks OK
(8)
multiple metrics
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['acc', 'mae'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Model architecture size: 1KB
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Model state (serialized) size: 0MB
CONTEXT: PL/Python function "madlib_keras_fit"
ERROR: plpy.SPIError: plpy.Error: Only at most one metric is supported.
(plpython.c:5038) (seg0 slice1 10.128.0.41:40000 pid=24904) (plpython.c:5038)
DETAIL:
Traceback (most recent call last):
PL/Python function "fit_transition", line 6, in <module>
return madlib_keras.fit_transition(**globals())
PL/Python function "fit_transition", line 430, in fit_transition
PL/Python function "fit_transition", line 101, in compile_and_set_weights
PL/Python function "fit_transition", line 288, in compile_model
PL/Python function "fit_transition", line 176, in
parse_and_validate_compile_params
PL/Python function "fit_transition", line 190, in _validate_metrics
PL/Python function "fit_transition", line 96, in _assert
PL/Python function "fit_transition"
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 198, in fit
PL/Python function "madlib_keras_fit"
{code}
Looks OK
(9)
no metrics
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam' $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1 ]-------------+----------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam'
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:07:28.589684
end_training_time | 2019-05-29 20:07:34.462567
metrics_elapsed_time | {5.87289786338806}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type |
training_metrics_final |
training_loss_final | 1.10092568398
training_metrics |
training_loss | {1.10092568397522}
validation_metrics_final |
validation_loss_final | 1.15330994129
validation_metrics |
validation_loss | {1.15330994129181}
metrics_iters | {3}
{code}
Looks OK, now left it blank (not 0 like before)
(10)
no fit param
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
NULL, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params |
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:09:15.684927
end_training_time | 2019-05-29 20:09:21.254505
metrics_elapsed_time | {5.56954598426819}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.316666662693
training_loss_final | 3.12867188454
training_metrics | {0.316666662693024}
training_loss | {3.12867188453674}
validation_metrics_final | 0.40000000596
validation_loss_final | 2.71730542183
validation_metrics | {0.400000005960464}
validation_loss | {2.71730542182922}
metrics_iters | {3}
{code}
Looks OK
(11)
no compile params
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
NULL, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
madlib(# 'iris_model', -- model
madlib(# 'model_arch_library', --
model_arch_table
madlib(# 1, -- model_arch_id
madlib(# NULL, -- compile_params
madlib(# $$ batch_size=16, epochs=1 $$, --
fit_params
madlib(# 3, -- num_iterations
madlib(# 0, -- gpus per host
madlib(# 'iris_test_packed', --
validation_table
madlib(# 3, -- metrics compute
frequency
madlib(# 'Sophia L.', -- name
madlib(# 'Simple MLP model on iris dataset' --
description
madlib(# );
ERROR: plpy.Error: Compile parameters cannot be empty or NULL.
(plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 42, in wrapper
PL/Python function "madlib_keras_fit", line 69, in fit
PL/Python function "madlib_keras_fit", line 96, in _assert
PL/Python function "madlib_keras_fit"
{code}
Looks OK, better error message
(12)
unsupported metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'top_k_categorical_accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
ERROR: plpy.SPIError: plpy.Error: Metric top_k_categorical_accuracy is not
supported. (plpython.c:5038) (seg0 slice1 10.128.0.41:40000 pid=16057)
(plpython.c:5038)
DETAIL:
Traceback (most recent call last):
PL/Python function "fit_transition", line 6, in <module>
return madlib_keras.fit_transition(**globals())
PL/Python function "fit_transition", line 452, in fit_transition
PL/Python function "fit_transition", line 101, in compile_and_set_weights
PL/Python function "fit_transition", line 299, in compile_model
PL/Python function "fit_transition", line 187, in
parse_and_validate_compile_params
PL/Python function "fit_transition", line 203, in _validate_metrics
PL/Python function "fit_transition", line 96, in _assert
PL/Python function "fit_transition"
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 42, in wrapper
PL/Python function "madlib_keras_fit", line 182, in fit
PL/Python function "madlib_keras_fit"
{code}
Looks OK.
(12)
same loss and metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'kullback_leibler_divergence'] $$, --
compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
1, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.82143497467 sec, Total was 1.03211283684
sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.701339960098 sec, Total was
0.910417079926 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 2.95784187317 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.375225782394 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 1: 1.98858952522.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 1: 1.98858952522.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.361536979675 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 1: 1.77411842346.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 1: 1.77411842346.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.893284082413 sec, Total was
1.19611501694 sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.903259992599 sec, Total was
1.19773983955 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 1.2005739212 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.342696905136 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 2: 1.88822972775.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 2: 1.88822972775.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.262344837189 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 2: 1.68969964981.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 2: 1.68969964981.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.699074983597 sec, Total was
0.957545042038 sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.71577501297 sec, Total was
0.968965053558 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.972018957138 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.28919506073 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 1.78998374939.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 1.78998374939.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.324784994125 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 1.61077010632.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 1.61077010632.
CONTEXT: PL/Python function "madlib_keras_fit"
-[ RECORD 1 ]----+-
madlib_keras_fit |
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+-------------------------------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'kullback_leibler_divergence']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 1
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 21:37:36.01207
end_training_time | 2019-05-29 21:37:43.27345
metrics_elapsed_time | {3.8685519695282,5.67480206489563,7.26137208938599}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {kullback_leibler_divergence}
training_metrics_final | 1.78998374939
training_loss_final | 1.78998374939
training_metrics | {1.98858952522278,1.88822972774506,1.78998374938965}
training_loss | {1.98858952522278,1.88822972774506,1.78998374938965}
validation_metrics_final | 1.61077010632
validation_loss_final | 1.61077010632
validation_metrics | {1.77411842346191,1.68969964981079,1.61077010631561}
validation_loss | {1.77411842346191,1.68969964981079,1.61077010631561}
metrics_iters | {1,2,3}
{code}
looks OK
was (Author: fmcquillan):
LGTM
Re-tested, all look OK except (4) which can be cleaned up in a later story.
(1)
What is the final list of metrics that are supported and those that are not
supported?
See:
https://www.pivotaltracker.com/n/projects/695473/stories/165776969/comments/202255072
https://docs.google.com/document/d/1RXC8bA4vZ3H3AmGbYtLHqi4fZ7WcMCVSXWNiSwRfZvo/edit
loss function not supported
'sparse_categorical_crossentropy'
metrics not supported
'sparse_categorical_accuracy'
'top_k_categorical_accuracy'
'sparse_top_k_categorical_accuracy'
(2)
fit with validation dataset, compute metrics at the end
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.54990196228 sec, Total was
0.758991956711 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.555626153946 sec, Total was
0.75802898407 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 2.43494582176 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.620418787003 sec, Total was
0.787483930588 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.660104036331 sec, Total was
0.832265853882 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 0.834777832031 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.574198007584 sec, Total was
0.753618955612 sec (seg0 slice1 10.128.0.41:40000 pid=26679)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.534913063049 sec, Total was
0.704334020615 sec (seg1 slice1 10.128.0.41:40001 pid=26680)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.756074905396 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.210503101349 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 0.324999988079.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 1.0396116972.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.190848112106 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 0.366666674614.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 1.06409192085.
CONTEXT: PL/Python function "madlib_keras_fit"
madlib_keras_fit
------------------
(1 row)
Time: 6817.188 ms
{code}
Verbose output: FIXED
INFO: Time for training in iteration 3: 0.756074905396 sec
INFO: Time for evaluation in iteration 3: 0.210503101349 sec.
Clarified training vs evaluation in INFO message: FIXED
(3)
fit with validation dataset, compute metrics at the end (con't)
{code}
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-23 22:33:08.611677
end_training_time | 2019-05-23 22:33:13.250892
metrics_elapsed_time | {4.63921880722046}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.324999988079
training_loss_final | 1.0396116972
training_metrics | {0.324999988079071}
training_loss | {1.03961169719696}
validation_metrics_final | 0.366666674614
validation_loss_final | 1.06409192085
validation_metrics | {0.366666674613953}
validation_loss | {1.06409192085266}
metrics_iters | {3}
{code}
Change to elapsed time looks better:
start_training_time | 2019-05-23 22:33:08.611677
end_training_time | 2019-05-23 22:33:13.250892
metrics_elapsed_time | {4.63921880722046}
13.250892 - 8.611677 = 4.639215
OK
(4)
fit with validation dataset, compute metrics every iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
1, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.535367012024 sec, Total was
0.746275901794 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.554301023483 sec, Total was
0.752547979355 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 3.40036702156 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.195158958435 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 1: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 1: 0.938332438469.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.332806825638 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 1: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 1: 0.944462954998.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.887582063675 sec, Total was
1.15123796463 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.860605955124 sec, Total was
1.12788796425 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 1.15389895439 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.192572116852 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 2: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 2: 0.900591552258.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.212724208832 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 2: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 2: 0.912341833115.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.607415914536 sec, Total was
0.850666046143 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.609027862549 sec, Total was
0.850383996964 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.852729082108 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.213857889175 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 0.865503013134.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.254362106323 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 0.666666686535.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 0.878823339939.
CONTEXT: PL/Python function "madlib_keras_fit"
-[ RECORD 1 ]----+-
madlib_keras_fit |
Time: 7409.327 ms
{code}
What is this saying?
INFO: Processed 60 images: Fit took 0.535367012024 sec, Total was
0.746275901794 sec (seg1 slice1 10.128.0.41:40001 pid=11506)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.554301023483 sec, Total was
0.752547979355 sec (seg0 slice1 10.128.0.41:40000 pid=11505)
Why does 'time for evaluation' repeat for every iteration and have different
times? e.g.,
INFO: Time for evaluation in iteration 3: 0.213857889175 sec.
INFO: Time for evaluation in iteration 3: 0.254362106323 sec.
{code}
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 1
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-23 22:38:33.342727
end_training_time | 2019-05-23 22:38:39.344795
metrics_elapsed_time | {3.45014500617981,4.83612895011902,6.00208592414856}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.358333319426
training_loss_final | 1.15425133705
training_metrics |
{0.358333319425583,0.358333319425583,0.358333319425583}
training_loss | {1.35349214076996,1.23959410190582,1.15425133705139}
validation_metrics_final | 0.233333334327
validation_loss_final | 1.31097722054
validation_metrics |
{0.233333334326744,0.233333334326744,0.233333334326744}
validation_loss | {1.57807862758636,1.42993664741516,1.31097722053528}
metrics_iters | {1,2,3}
Time intervals look OK:
metrics_elapsed_time = start_training_time - start_training_time
6.00208592414856 = 22:38:39.344795 - 22:38:33.342727
(5)
fit with validation dataset, compute metrics at different iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
2, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 2
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:01:42.08684
end_training_time | 2019-05-29 20:01:48.658938
metrics_elapsed_time | {5.42638921737671,6.57209706306458}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.341666668653
training_loss_final | 1.35804212093
training_metrics | {0.349999994039536,0.341666668653488}
training_loss | {1.42269325256348,1.35804212093353}
validation_metrics_final | 0.266666680574
validation_loss_final | 1.50677502155
validation_metrics | {0.266666680574417,0.266666680574417}
validation_loss | {1.58999609947205,1.50677502155304}
metrics_iters | {2,3}
Time: 22.104 ms
{code}
Looks OK {2,3}
(6)
fit with no validation dataset, compute metrics at different iteration
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
NULL, -- validation_table
2, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table |
metrics_compute_frequency | 2
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:02:54.856595
end_training_time | 2019-05-29 20:03:01.289986
metrics_elapsed_time | {4.89847302436829,6.4333918094635}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.316666662693
training_loss_final | 2.19523310661
training_metrics | {0.316666662693024,0.316666662693024}
training_loss | {2.31425333023071,2.19523310661316}
validation_metrics_final |
validation_loss_final |
validation_metrics |
validation_loss |
metrics_iters | {2,3}
{code}
Looks OK
(7)
a different metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['mae'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+---------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['mae']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:05:39.270399
end_training_time | 2019-05-29 20:05:42.27942
metrics_elapsed_time | {3.00901985168457}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {mae}
training_metrics_final | 0.46932977438
training_loss_final | 1.41450297832
training_metrics | {0.46932977437973}
training_loss | {1.41450297832489}
validation_metrics_final | 0.461133390665
validation_loss_final | 1.38406682014
validation_metrics | {0.461133390665054}
validation_loss | {1.38406682014465}
metrics_iters | {3}
{code}
Looks OK
(8)
multiple metrics
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['acc', 'mae'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Model architecture size: 1KB
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Model state (serialized) size: 0MB
CONTEXT: PL/Python function "madlib_keras_fit"
ERROR: plpy.SPIError: plpy.Error: Only at most one metric is supported.
(plpython.c:5038) (seg0 slice1 10.128.0.41:40000 pid=24904) (plpython.c:5038)
DETAIL:
Traceback (most recent call last):
PL/Python function "fit_transition", line 6, in <module>
return madlib_keras.fit_transition(**globals())
PL/Python function "fit_transition", line 430, in fit_transition
PL/Python function "fit_transition", line 101, in compile_and_set_weights
PL/Python function "fit_transition", line 288, in compile_model
PL/Python function "fit_transition", line 176, in
parse_and_validate_compile_params
PL/Python function "fit_transition", line 190, in _validate_metrics
PL/Python function "fit_transition", line 96, in _assert
PL/Python function "fit_transition"
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 198, in fit
PL/Python function "madlib_keras_fit"
{code}
Looks OK
(9)
no metrics
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam' $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# select * from iris_model_summary;
-[ RECORD 1 ]-------------+----------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam'
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:07:28.589684
end_training_time | 2019-05-29 20:07:34.462567
metrics_elapsed_time | {5.87289786338806}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type |
training_metrics_final |
training_loss_final | 1.10092568398
training_metrics |
training_loss | {1.10092568397522}
validation_metrics_final |
validation_loss_final | 1.15330994129
validation_metrics |
validation_loss | {1.15330994129181}
metrics_iters | {3}
{code}
Looks OK, now left it blank (not 0 like before)
(10)
no fit param
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='categorical_crossentropy',
optimizer='adam', metrics=['accuracy'] $$, -- compile_params
NULL, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
-[ RECORD 1
]-------------+--------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='categorical_crossentropy', optimizer='adam',
metrics=['accuracy']
fit_params |
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 3
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 20:09:15.684927
end_training_time | 2019-05-29 20:09:21.254505
metrics_elapsed_time | {5.56954598426819}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {accuracy}
training_metrics_final | 0.316666662693
training_loss_final | 3.12867188454
training_metrics | {0.316666662693024}
training_loss | {3.12867188453674}
validation_metrics_final | 0.40000000596
validation_loss_final | 2.71730542183
validation_metrics | {0.400000005960464}
validation_loss | {2.71730542182922}
metrics_iters | {3}
{code}
Looks OK
(11)
no compile params
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
NULL, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
madlib=# SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
madlib(# 'iris_model', -- model
madlib(# 'model_arch_library', --
model_arch_table
madlib(# 1, -- model_arch_id
madlib(# NULL, -- compile_params
madlib(# $$ batch_size=16, epochs=1 $$, --
fit_params
madlib(# 3, -- num_iterations
madlib(# 0, -- gpus per host
madlib(# 'iris_test_packed', --
validation_table
madlib(# 3, -- metrics compute
frequency
madlib(# 'Sophia L.', -- name
madlib(# 'Simple MLP model on iris dataset' --
description
madlib(# );
ERROR: plpy.Error: Compile parameters cannot be empty or NULL.
(plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 42, in wrapper
PL/Python function "madlib_keras_fit", line 69, in fit
PL/Python function "madlib_keras_fit", line 96, in _assert
PL/Python function "madlib_keras_fit"
{code}
Looks OK, better error message
(12)
unsupported metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'top_k_categorical_accuracy'] $$, -- compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
3, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
ERROR: plpy.SPIError: plpy.Error: Metric top_k_categorical_accuracy is not
supported. (plpython.c:5038) (seg0 slice1 10.128.0.41:40000 pid=16057)
(plpython.c:5038)
DETAIL:
Traceback (most recent call last):
PL/Python function "fit_transition", line 6, in <module>
return madlib_keras.fit_transition(**globals())
PL/Python function "fit_transition", line 452, in fit_transition
PL/Python function "fit_transition", line 101, in compile_and_set_weights
PL/Python function "fit_transition", line 299, in compile_model
PL/Python function "fit_transition", line 187, in
parse_and_validate_compile_params
PL/Python function "fit_transition", line 203, in _validate_metrics
PL/Python function "fit_transition", line 96, in _assert
PL/Python function "fit_transition"
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_fit", line 21, in <module>
madlib_keras.fit(**globals())
PL/Python function "madlib_keras_fit", line 42, in wrapper
PL/Python function "madlib_keras_fit", line 182, in fit
PL/Python function "madlib_keras_fit"
{code}
Looks OK.
(12)
same loss and metric
{code}
DROP TABLE IF EXISTS iris_model, iris_model_summary;
SELECT madlib.madlib_keras_fit('iris_train_packed', -- source_table
'iris_model', -- model
'model_arch_library', -- model_arch_table
1, -- model_arch_id
$$ loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'kullback_leibler_divergence'] $$, --
compile_params
$$ batch_size=16, epochs=1 $$, -- fit_params
3, -- num_iterations
0, -- gpus per host
'iris_test_packed', -- validation_table
1, -- metrics compute frequency
'Sophia L.', -- name
'Simple MLP model on iris dataset' --
description
);
INFO: Processed 60 images: Fit took 0.82143497467 sec, Total was 1.03211283684
sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.701339960098 sec, Total was
0.910417079926 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 1: 2.95784187317 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.375225782394 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 1: 1.98858952522.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 1: 1.98858952522.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 1: 0.361536979675 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 1: 1.77411842346.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 1: 1.77411842346.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.893284082413 sec, Total was
1.19611501694 sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.903259992599 sec, Total was
1.19773983955 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 2: 1.2005739212 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.342696905136 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 2: 1.88822972775.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 2: 1.88822972775.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 2: 0.262344837189 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 2: 1.68969964981.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 2: 1.68969964981.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Processed 60 images: Fit took 0.699074983597 sec, Total was
0.957545042038 sec (seg0 slice1 10.128.0.41:40000 pid=18434)
CONTEXT: PL/Python function "fit_transition"
INFO: Processed 60 images: Fit took 0.71577501297 sec, Total was
0.968965053558 sec (seg1 slice1 10.128.0.41:40001 pid=18435)
CONTEXT: PL/Python function "fit_transition"
INFO: Time for training in iteration 3: 0.972018957138 sec
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.28919506073 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set metric after iteration 3: 1.78998374939.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Training set loss after iteration 3: 1.78998374939.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Time for evaluation in iteration 3: 0.324784994125 sec.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set metric after iteration 3: 1.61077010632.
CONTEXT: PL/Python function "madlib_keras_fit"
INFO: Validation set loss after iteration 3: 1.61077010632.
CONTEXT: PL/Python function "madlib_keras_fit"
-[ RECORD 1 ]----+-
madlib_keras_fit |
madlib=# select * from iris_model_summary;
-[ RECORD 1
]-------------+-------------------------------------------------------------------------------------------------
source_table | iris_train_packed
model | iris_model
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
model_arch_id | 1
compile_params | loss='kullback_leibler_divergence',
optimizer='adam', metrics=[ 'kullback_leibler_divergence']
fit_params | batch_size=16, epochs=1
num_iterations | 3
validation_table | iris_test_packed
metrics_compute_frequency | 1
name | Sophia L.
description | Simple MLP model on iris dataset
model_type | madlib_keras
model_size | 0.7939453125
start_training_time | 2019-05-29 21:37:36.01207
end_training_time | 2019-05-29 21:37:43.27345
metrics_elapsed_time | {3.8685519695282,5.67480206489563,7.26137208938599}
madlib_version | 1.16-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
metrics_type | {kullback_leibler_divergence}
training_metrics_final | 1.78998374939
training_loss_final | 1.78998374939
training_metrics | {1.98858952522278,1.88822972774506,1.78998374938965}
training_loss | {1.98858952522278,1.88822972774506,1.78998374938965}
validation_metrics_final | 1.61077010632
validation_loss_final | 1.61077010632
validation_metrics | {1.77411842346191,1.68969964981079,1.61077010631561}
validation_loss | {1.77411842346191,1.68969964981079,1.61077010631561}
metrics_iters | {1,2,3}
{code}
looks OK
> DL: Add support for reporting various metrics in fit/evaluate
> -------------------------------------------------------------
>
> Key: MADLIB-1338
> URL: https://issues.apache.org/jira/browse/MADLIB-1338
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Deep Learning
> Reporter: Nandish Jayaram
> Priority: Major
> Fix For: v1.16
>
>
> The current `madlib_keras.fit()` code reports accuracy as the only metric,
> along with loss value. But we could ask for different metrics in compile
> params (`mae`, `binary_accuracy ` etc.), then `Keras.evaluate()` would return
> back `loss` (by default) and `mean_absolute_error` or `binary_accuracy`
> (metrics).
> This JIRA requests support to be able to report any one of these metrics in
> the output table.
> Other requirements:
> 1. Remove training loss/accuracy computation from `fit_transition` and
> instead use the evaluate function to calculate the training loss/metric. See
> PR [https://github.com/apache/madlib/pull/388
> |https://github.com/apache/madlib/pull/388/files]for more details
> 2. metric param can be optional
> 3. Maybe we should rename all the related output column as metric instead of
> metrics
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
