kaknikhil commented on a change in pull request #425: DL: Add training for
multiple models
URL: https://github.com/apache/madlib/pull/425#discussion_r310256081
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras.py_in
##########
@@ -589,31 +637,49 @@ def get_loss_metric_from_keras_eval(schema_madlib,
table, compile_params,
def internal_keras_eval_transition(state, dependent_var, independent_var,
model_architecture, serialized_weights,
compile_params,
current_seg_id, seg_ids, images_per_seg,
- gpus_per_host, segments_per_host, **kwargs):
+ gpus_per_host, segments_per_host,
+ is_final, **kwargs):
SD = kwargs['SD']
device_name = get_device_name_and_set_cuda_env(gpus_per_host,
current_seg_id)
agg_loss, agg_metric, agg_image_count = state
- if not agg_image_count:
- set_keras_session(device_name, gpus_per_host, segments_per_host)
- model = model_from_json(model_architecture)
- compile_and_set_weights(model, compile_params, device_name,
- serialized_weights)
+ # User called evaluate will always set is_final to true.
+ # If is_final is false, that means the fit already created a session and a
graph
+ # Otherwise, we may (last iteration of fit) or may not (user evaluate call)
+ # have a session.
+ if is_final and 'sess' not in SD:
+ sess = get_keras_session(device_name, gpus_per_host, segments_per_host)
+ SD['sess'] = sess
+ K.set_session(sess)
+ # Popping the segment model kept in the SD of
internal_keras_eval_transition,
+ # which is leftover from the previous iteration. But the session is
already
+ # cleared by fit_transition at this time, so the model cannot be
re-used.
+ SD.pop('segment_model', None)
Review comment:
Since evaluate is a different function than fit, SD will prob be reset for
the evaluate function. But we can still remove the key just to be safe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
