makemebitter commented on a change in pull request #425: DL: Add training for
multiple models
URL: https://github.com/apache/madlib/pull/425#discussion_r311269249
##########
File path: src/ports/postgres/modules/deep_learning/madlib_keras.py_in
##########
@@ -455,17 +495,23 @@ def fit_transition(state, dependent_var,
independent_var, model_architecture,
# Once done with all images on a segment, we update weights
# with the total number of images here instead of the merge function.
# The merge function only deals with aggregating them.
- updated_weights = [ total_images * w for w in updated_weights ]
- # In GPDB, each segment would have a keras session, so clear
- # them after the last buffer is processed.
- clear_keras_session()
-
+ if not is_cerebro:
+ updated_weights = [ total_images * w for w in updated_weights ]
+
+ # In GPDB, each segment would have a keras session, so clear
+ # them at the end of final iteration and also every final buffer for
cerebro
+ if is_final or is_cerebro:
+ K.clear_session()
Review comment:
This was intended to maintain SD['sess'] explicitly. For now, I think we can
call clear_keras_session() to clear and close the session. I'm less inclined to
include the destruction of SD here, as this function is also called from
multiple places where no SD deletion needed there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
