fmcquillan99 edited a comment on pull request #518:
URL: https://github.com/apache/madlib/pull/518#issuecomment-698586366
(4)
`hyperopt` is not learning in the tests I have done so far with `rand`
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr':
[0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr':
[0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr':
[0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs':
[1]}$$,
'hyperopt',
'num_models=3, num_iters=5,
algorithm=rand',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute
freq
NULL, -- name
NULL); -- descr
```
produces no-learning result with labels:
```
INFO: ***Evaluating 3 newly suggested model configurations***
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 659.287541866 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0988999977708, loss=14.5240154266
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 2: 663.107329845 sec
DETAIL:
Training set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
Validation set after iteration 2:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0989999994636, loss=14.5224027634
mst_key=2: metric=0.10000000149, loss=nan
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
When I run `hyperband` with same models/hyperparams, it does learn:
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr':
[0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr':
[0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr':
[0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs':
[1]}$$,
'hyperband',
'R=6, eta=3, skip_last=0',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute
freq
NULL, -- name
NULL); -- descr
```
produces
```
PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 655.647205114 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.34889999032, loss=1.83796536922
mst_key=1: metric=0.468120008707, loss=1.481554389
mst_key=2: metric=0.475380003452, loss=1.63904440403
Validation set after iteration 1:
mst_key=3: metric=0.355800002813, loss=1.83082151413
mst_key=1: metric=0.470499992371, loss=1.47972774506
mst_key=2: metric=0.470200002193, loss=1.66988003254
CONTEXT: PL/Python function "madlib_keras_automl"
etc.
```
Also `hyperopt` with `tpe` is not learning cifar10
```
SELECT madlib.madlib_keras_automl('cifar10_train_packed',
'automl_cifar10_output',
'model_arch_library',
'automl_cifar10_mst_table',
ARRAY[1,2,3],
$${'loss': ['categorical_crossentropy'],
'optimizer_params_list': [
{'optimizer': ['Adam'],'lr':
[0.0001, 0.01, 'log']},
{'optimizer': ['RMSprop'],'lr':
[0.0001, 0.01, 'log'], 'decay': [1.0e-7, 1.0e-5, 'log']},
{'optimizer': ['SGD'],'lr':
[0.0001, 0.01, 'log'], 'momentum': [0.9, 0.99,'log']}],
'metrics':['accuracy']}$$,
$${'batch_size': [64,128], 'epochs':
[1]}$$,
'hyperopt',
'num_models=3, num_iters=5, algorithm=tpe',
NULL, -- random state
NULL, -- object table
FALSE, -- use GPUs
'cifar10_val_packed', -- validation table
1, -- metrics compute
freq
NULL, -- name
NULL); -- descr
```
produces
```
CONTEXT: PL/Python function "madlib_keras_automl"
INFO:
Time for training in iteration 1: 648.608314991 sec
DETAIL:
Training set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.0999400019646, loss=14.5072526932
mst_key=2: metric=0.10000000149, loss=14.5062847137
Validation set after iteration 1:
mst_key=3: metric=0.10000000149, loss=nan
mst_key=1: metric=0.10000000149, loss=14.5062856674
mst_key=2: metric=0.10000000149, loss=14.5062847137
CONTEXT: PL/Python function "madlib_keras_automl"
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
