[
https://issues.apache.org/jira/browse/MADLIB-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831872#comment-16831872
]
Nandish Jayaram commented on MADLIB-1335:
-----------------------------------------
*Expected columns in output summary table*:
# {{metrics_compute_frequency}}, TEXT: The input param that was passed to fit.
# {{training_loss_final}}, DOUBLE PRECISION: Final iteration's training loss.
# {{training_metrics_final}} DOUBLE PRECISION []: Final iteration's training
metrics.
# {{validation_loss_final}}, DOUBLE PRECISION: Final iteration's validation
loss.
# {{validation_metrics_final}} DOUBLE PRECISION []: Final iteration's
validation metrics.
# {{training_loss}} DOUBLE PRECISION []: training loss at various iterations
determined using {{metrics_compute_frequency}}param.
# {{training_metrics}} DOUBLE PRECISION []: training metrics at various
iterations determined using {{metrics_compute_frequency}} param. Note that this
is a 2-D array, where each inner dimension captures all metrics (even if it's
just one) for a given iteration.
# {{validation_loss}} DOUBLE PRECISION []: validation loss at various
iterations determined using {{metrics_compute_frequency}} param.
# {{validation_metrics}} DOUBLE PRECISION []: validation metrics at various
iterations determined using {{metrics_compute_frequency}} param. Note that this
is a 2-D array, where each inner dimension captures all metrics (even if it's
just one) for a given iteration.
# {{metrics_iters}} DOUBLE PRECISION []: Array containing the various
iteration numbers at which loss/metrics were computed for training/validation
data.
*NOTE*
# For this JIRA, limit the scope of {{metrics}} to only {{accuracy}}. Other
metrics support to be handled in
https://issues.apache.org/jira/browse/MADLIB-1338.
# {{metrics_compute_frequency}} is an optional param and the default value is
NULL. If NULL, then for loss/accuracy over training/validation, only report the
{{*_final}} values, and all the iterations related values would be NULL.
> Add new param metrics_compute_frequency to madlib_keras_fit()
> -------------------------------------------------------------
>
> Key: MADLIB-1335
> URL: https://issues.apache.org/jira/browse/MADLIB-1335
> Project: Apache MADlib
> Issue Type: New Feature
> Components: Deep Learning
> Reporter: Ekta Khanna
> Priority: Major
> Fix For: v1.16
>
>
> JIRA: https://issues.apache.org/jira/browse/MADLIB-1335
> Context
> Getting per iteration loss and other metrics for training and validation data
> sets can be expensive. This parameter is intended to give control to user on
> how often to do this computation.
> Story
> As a data scientist, I want to specify how often to calculate loss and other
> metrics for training and validation data (if provided), i.e, every n
> iterations
> Interface
> {code}
> madlib_keras_fit(
> source_table VARCHAR,
> model VARCHAR,
> dependent_varname VARCHAR,
> independent_varname VARCHAR,
> model_arch_table VARCHAR,
> model_arch_id INTEGER,
> compile_params VARCHAR,
> fit_params VARCHAR,
> num_iterations INTEGER,
> use_gpu BOOLEAN,
> validation_table VARCHAR,
> metrics_compute_frequency INTEGER <-------- NEW OPTIONAL PARAM
> name VARCHAR,
> description VARCHAR
> {code}
> where `metrics_compute_frequency` is an optional parameter that means:
> {code}
> NULL - calculate loss and metrics only on final model after last iteration
> (default)
> n - calculate loss metrics every n-th iteration and on final model after last
> iteration
> {code}
> and `metrics_compute_frequency must be >=1 and <=num_iterations`
> Acceptance
> 1) Set `num_iterations = 12` and leave ` metrics_compute_frequency` as
> default and get 1 loss and metrics value at end after 12 iterations
> 2) Set `num_iterations = 12` and set `metrics_compute_frequency = 5` and get
> loss and metrics after 5th and 10th iterations and at end after 12 iterations
> 3) Set `metrics_compute_frequency = 0` and get an error
> 4) Set `metrics_compute_frequency = num_iterations+1` and get an error
> 5) Test with different metrics: from https://keras.io/metrics/ try 'mae' ,
> 'acc' etc.
> Reference
> [1] https://keras.io/metrics/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
