fmcquillan99 commented on issue #395: DL: madlib_keras_evaluate() function URL: https://github.com/apache/madlib/pull/395#issuecomment-497890456 Please have a look at (5) below: (0) interface ``` CREATE OR REPLACE FUNCTION MADLIB_SCHEMA.madlib_keras_evaluate( model_table VARCHAR, test_table VARCHAR, output_table VARCHAR, gpus_per_host INTEGER ``` (1) train a model ``` DROP TABLE IF EXISTS iris_model, iris_model_summary; SELECT madlib.madlib_keras_fit('iris_train_packed', -- source table 'iris_model', -- model output table 'model_arch_library', -- model arch table 1, -- model arch id $$ loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'] $$, -- compile_params $$ batch_size=5, epochs=3 $$, -- fit_params 20 -- num_iterations ); DROP TABLE IF EXISTS iris_model, iris_model_summary; SELECT madlib.madlib_keras_fit('iris_train_packed', -- source table 'iris_model', -- model output table 'model_arch_library', -- model arch table 1, -- model arch id $$ loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'] $$, -- compile_params $$ batch_size=5, epochs=3 $$, -- fit_params 20, -- num_iterations 0, -- GPUs per host 'iris_test_packed' -- validation dataset ); INFO: Training set metric after iteration 20: 0.975000023842. CONTEXT: PL/Python function "madlib_keras_fit" INFO: Training set loss after iteration 20: 0.222917750478. CONTEXT: PL/Python function "madlib_keras_fit" INFO: Validation set metric after iteration 20: 0.966666638851. CONTEXT: PL/Python function "madlib_keras_fit" INFO: Validation set loss after iteration 20: 0.176595017314. CONTEXT: PL/Python function "madlib_keras_fit" ``` (2) evaluate the training set ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_model', -- model 'iris_train_packed', -- training table 'iris_validate' -- output table ); SELECT * FROM iris_validate; loss | metric | metrics_type -------------------+-------------------+-------------- 0.222917750477791 | 0.975000023841858 | {accuracy} (1 row) ``` OK, evaluate loss and accuracy match the fit now manually count accuracy via predict: ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_model', -- model 'iris_train', -- train_table 'id', -- id column 'attributes', -- independent var 'iris_predict' -- output table ); SELECT round(count(*)*100/(150.0*0.8),2) as train_accuracy_percent from (select iris_train.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_train on iris_train.id=iris_predict.id) q WHERE q.actual=q.estimated; train_accuracy_percent ------------------------ 97.50 ``` OK, evaluate loss and accuracy match predict (3) evaluate the test set ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_model', -- model 'iris_test_packed', -- training table 'iris_validate' -- output table ); loss | metric | metrics_type -------------------+-------------------+-------------- 0.176595017313957 | 0.966666638851166 | {accuracy} (1 row) ``` OK, evaluate loss and accuracy match the fit above now manually count accuracy via predict: ``` DROP TABLE IF EXISTS iris_predict; SELECT madlib.madlib_keras_predict('iris_model', -- model 'iris_test', -- test_table 'id', -- id column 'attributes', -- independent var 'iris_predict' -- output table ); SELECT round(count(*)*100/(150.0*0.2),2) as test_accuracy_percent from (select iris_test.class_text as actual, iris_predict.estimated_class_text as estimated from iris_predict inner join iris_test on iris_test.id=iris_predict.id) q WHERE q.actual=q.estimated; test_accuracy_percent ----------------------- 96.67 (1 row) ``` OK, evaluate loss and accuracy match predict (4) pass in table that has not been mini-batched ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_model', -- model 'iris_test', -- test table 'iris_validate' -- output table ); WARNING: column "independent_var" does not exist CONTEXT: PL/Python function "madlib_keras_evaluate" ERROR: plpy.Error: madlib_keras_evaluate error: invalid independent_varname ('independent_var') for test table (iris_test). (plpython.c:5038) CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 547, in evaluate PL/Python function "madlib_keras_evaluate", line 155, in __init__ PL/Python function "madlib_keras_evaluate", line 99, in __init__ PL/Python function "madlib_keras_evaluate", line 158, in _validate_input_args PL/Python function "madlib_keras_evaluate", line 107, in _validate_input_args PL/Python function "madlib_keras_evaluate", line 132, in _validate_test_tbl_cols PL/Python function "madlib_keras_evaluate", line 96, in _assert PL/Python function "madlib_keras_evaluate" ``` This will be a common mistake, I'm not sure if the message above is the best. (5) ask for gpu when none avail ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_model', -- model 'iris_test_packed', -- training table 'iris_validate', -- output table 1 -- gpus per host ); WARNING: The number of gpus per host is less than the number of segments per host. The support for this case is experimental and it may fail. CONTEXT: PL/Python function "madlib_keras_evaluate" ERROR: plpy.SPIError: tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Adam/iterations: node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device. The requested device appears to be a GPU, but CUDA is not enabled. (plpython.c:5038) (seg1 slice1 10.128.0.41:40001 pid=20148) (plpython.c:5038) DETAIL: [[node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) ]] Caused by op u'Adam/iterations', defined at: File "<string>", line 1, in <module> File "<string>", line 7, in __plpython_procedure_internal_keras_eval_transition_110195 File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras.py", line 644, in internal_keras_eval_transition serialized_weights) File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras_wrapper.py", line 109, in compile_and_set_weights compile_model(segment_model, compile_params) File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras_wrapper.py", line 309, in compile_model model.compile(**compile_dict) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/engine/training.py", line 96, in compile self.optimizer = optimizers.get(optimizer) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 796, in get return deserialize(config) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 768, in deserialize printable_module_name='optimizer') File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 147, in deserialize_keras_object return cls.from_config(config['config']) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 154, in from_config return cls(**config) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 462, in __init__ self.iterations = K.variable(0, dtype='int64', name='iterations') File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 402, in variable v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 213, in __call__ return cls._variable_v1_call(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 176, in _variable_v1_call aggregation=aggregation) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 155, in <lambda> previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2495, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 217, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 1395, in __init__ constraint=constraint) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 1531, in _init_from_args name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/state_ops.py", line 79, in variable_op_v2 shared_name=shared_name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 1425, in variable_v2 shared_name=shared_name, name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() InvalidArgumentError (see above for traceback): Cannot assign a device for operation Adam/iterations: node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device. The requested device appears to be a GPU, but CUDA is not enabled. [[node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) ]] Traceback (most recent call last): PL/Python function "internal_keras_eval_transition", line 6, in <module> return madlib_keras.internal_keras_eval_transition(**globals()) PL/Python function "internal_keras_eval_transition", line 643, in internal_keras_eval_transition PL/Python function "internal_keras_eval_transition", line 111, in compile_and_set_weights PL/Python function "internal_keras_eval_transition", line 507, in set_weights PL/Python function "internal_keras_eval_transition", line 2469, in batch_set_value PL/Python function "internal_keras_eval_transition", line 198, in get_session PL/Python function "internal_keras_eval_transition", line 928, in run PL/Python function "internal_keras_eval_transition", line 1151, in _run PL/Python function "internal_keras_eval_transition", line 1327, in _do_run PL/Python function "internal_keras_eval_transition", line 1347, in _do_call PL/Python function "internal_keras_eval_transition" CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 582, in evaluate PL/Python function "madlib_keras_evaluate", line 625, in get_loss_metric_from_keras_eval PL/Python function "madlib_keras_evaluate" ``` Is this ^^^ the right message to throw and stack trace that we want? ``` DROP TABLE IF EXISTS iris_validate; SELECT madlib.madlib_keras_evaluate('iris_model', -- model 'iris_test_packed', -- training table 'iris_validate', -- output table 5 -- gpus per host ); ERROR: plpy.SPIError: tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation Adam/iterations: node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device. The requested device appears to be a GPU, but CUDA is not enabled. (plpython.c:5038) (seg0 slice1 10.128.0.41:40000 pid=20691) (plpython.c:5038) DETAIL: [[node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) ]] Caused by op u'Adam/iterations', defined at: File "<string>", line 1, in <module> File "<string>", line 7, in __plpython_procedure_internal_keras_eval_transition_110195 File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras.py", line 644, in internal_keras_eval_transition serialized_weights) File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras_wrapper.py", line 109, in compile_and_set_weights compile_model(segment_model, compile_params) File "/home/gpadmin/madlib/build/src/ports/greenplum/5/modules/deep_learning/madlib_keras_wrapper.py", line 309, in compile_model model.compile(**compile_dict) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/engine/training.py", line 96, in compile self.optimizer = optimizers.get(optimizer) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 796, in get return deserialize(config) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 768, in deserialize printable_module_name='optimizer') File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 147, in deserialize_keras_object return cls.from_config(config['config']) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 154, in from_config return cls(**config) File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/optimizers.py", line 462, in __init__ self.iterations = K.variable(0, dtype='int64', name='iterations') File "/home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 402, in variable v = tf.Variable(value, dtype=tf.as_dtype(dtype), name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 213, in __call__ return cls._variable_v1_call(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 176, in _variable_v1_call aggregation=aggregation) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 155, in <lambda> previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variable_scope.py", line 2495, in default_variable_creator expected_shape=expected_shape, import_scope=import_scope) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 217, in __call__ return super(VariableMetaclass, cls).__call__(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 1395, in __init__ constraint=constraint) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/variables.py", line 1531, in _init_from_args name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/state_ops.py", line 79, in variable_op_v2 shared_name=shared_name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 1425, in variable_v2 shared_name=shared_name, name=name) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper op_def=op_def) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func return func(*args, **kwargs) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3300, in create_op op_def=op_def) File "/home/gpadmin/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1801, in __init__ self._traceback = tf_stack.extract_stack() InvalidArgumentError (see above for traceback): Cannot assign a device for operation Adam/iterations: node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:XLA_CPU:0 ]. Make sure the device specification refers to a valid device. The requested device appears to be a GPU, but CUDA is not enabled. [[node Adam/iterations (defined at /home/gpadmin/.local/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py:402) ]] Traceback (most recent call last): PL/Python function "internal_keras_eval_transition", line 6, in <module> return madlib_keras.internal_keras_eval_transition(**globals()) PL/Python function "internal_keras_eval_transition", line 643, in internal_keras_eval_transition PL/Python function "internal_keras_eval_transition", line 111, in compile_and_set_weights PL/Python function "internal_keras_eval_transition", line 507, in set_weights PL/Python function "internal_keras_eval_transition", line 2469, in batch_set_value PL/Python function "internal_keras_eval_transition", line 198, in get_session PL/Python function "internal_keras_eval_transition", line 928, in run PL/Python function "internal_keras_eval_transition", line 1151, in _run PL/Python function "internal_keras_eval_transition", line 1327, in _do_run PL/Python function "internal_keras_eval_transition", line 1347, in _do_call PL/Python function "internal_keras_eval_transition" CONTEXT: Traceback (most recent call last): PL/Python function "madlib_keras_evaluate", line 21, in <module> return madlib_keras.evaluate(**globals()) PL/Python function "madlib_keras_evaluate", line 582, in evaluate PL/Python function "madlib_keras_evaluate", line 625, in get_loss_metric_from_keras_eval PL/Python function "madlib_keras_evaluate" ``` Is this ^^^ the right message to throw and stack trace that we want?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
