[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168293#comment-17168293
]
Wangda Tan edited comment on SUBMARINE-548 at 7/30/20, 11:22 PM:
-
[~jotjohnting], thanks for working on this, I just reviewed
[https://github.com/apache/submarine/pull/351]
I think we missed some part in the design:
The design doc:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
defined the spec of how to submit a pre-defined template, which will be
sufficient for submission from CLI/REST/UI. However, it is not enough to
*register/define* a pre-defined template.
The differences between register and submission a pre-defined template are:
* *Register* an experiment-template requires information of how Submarine can
run the experiment, for example, it needs to include: resources required for
worker; environment (docker image, conda kernel); commandline options for
workers/ps, etc.
* In contrast, *submit* an experiment-template only requires filling
required/optional parameters.
So to register a pre-defined template, we need to *not only* include
ExperimentTemplate, but also, we need to tell how Submarine can run it.
*So the predefined template registration should include the following:*
*1) A template of Experiment yaml, for example, if we take an experiment
example from our* doc:
[https://github.com/apache/submarine/blob/master/docs/userdocs/k8s/run-tensorflow-experiment.md]
{code:java}
meta:
name: "tf-mnist-yaml"
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --log_dir=/train/log
--learning_rate=0.01 --batch_size=150"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
We can create a template of the YAML (with placeholders) using syntax like:
{code:java}
meta:
name: {{name}}
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --input {{input}}
--log_dir=/train/log --learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M" {code}
The above template defined 3 variables (placeholders):
* name
* input
* training.learning_rate.
* training.batch_size
(The above YAML placeholder is based on [https://stackoverflow.com/a/41620747)]
*2) A list of parameters (Similar to ExperimentTemplate)*
*So I think we need the following object:*
*a. RegisterExperimentTemplateSpec*
{code:java}
{
template_name: Name of the template
experiment_spec: the spec for experiment with placeholders.
parameters:
List of parameters definition
} {code}
*a. SubmissionExperimentTemplateSpec*
{code:java}
{
experiment_name: Name of the running experiment
template_name: Name of the template
parameters:
List of parameters (with values)
} {code}
Does this make sense? cc: [~pingsutw], [~ztang] for suggestions.
And if we agree with the proposal, we need to update our experiment spec design
accordingly:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
was (Author: wangda):
[~jotjohnting], thanks for working on this, I just reviewed
[https://github.com/apache/submarine/pull/351]
I think we missed some part in the design:
The design doc:
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
defined the spec of how to submit a pre-defined template, which will be
sufficient for submission from CLI/REST/UI. However, it is not enough to
*register/define* a pre-defined template.
The differences between register and submission a pre-defined template are:
* *Register* an experiment-template requires information of how Submarine can
run the experiment, for example, it needs to include: resources required for
worker; environment (docker image, conda kernel); commandline options for
workers/ps, etc.
* In contrast, *submit* an experiment-template only requires filling
required/optional parameters.
So to register a pre-defined template, we need to *not only* include
ExperimentTemplate, but also, we need to tell how Submarine can run it.
*So the predefined template registration should include the following:*
*1) A template of Experiment yaml, for example, if we take an experiment
example from our* doc:
[https://github.com/
[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169515#comment-17169515
]
Kevin Su edited comment on SUBMARINE-548 at 8/2/20, 11:22 AM:
--
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description: >
Train data is expected in SVM format, and can be stored in HDFS/S3
name: training.learning_rate
required: true
default: 0.001
description: >
Learning rate for mnist model, default is 0.001
name: training.batch_size
required: true
description: >
Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name:
mnist_example
template_name:
mnist_template
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
}{code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
was (Author: pingsutw):
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{
template_name: mnist_template
experiment_spec:
meta:
name: \{{name}}
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py --input
\{{input.train_data}} --log_dir=/train/log --
learning_rate=\{{training.learning_rate}}
--batch_size=\{{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
- name: input.train_data
required: true
description: >
Train data is expected in SVM format, and can be stored in HDFS/S3
- name: training.learning_rate
required: true
description: >
Learning rate for mnist model, default is 0.001
- name: training.batch_size
required: true
description: >
Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32
}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{
experiment_name: mnist_example
template_name: mnist_template
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
> [Umbrella] Predefined Experiment
>
>
> Key: SUBMARINE-548
> URL: https://issues.apache.org/jira/browse/SUBMARINE-548
> Project: Apache Submarine
> Issue Type: New Feature
> Components: experiment templ
[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169515#comment-17169515
]
Kevin Su edited comment on SUBMARINE-548 at 8/2/20, 11:23 AM:
--
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description:
"Train data is expected in SVM format, and can be stored in HDFS/S3"
name: training.learning_rate
required: true
default: 0.001
description:
"Learning rate for mnist model, default is 0.001"
name: training.batch_size
required: true
description:
"Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32"
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name:
mnist_example
template_name:
mnist_template
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
} {code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
was (Author: pingsutw):
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description: >
Train data is expected in SVM format, and can be stored in HDFS/S3
name: training.learning_rate
required: true
default: 0.001
description: >
Learning rate for mnist model, default is 0.001
name: training.batch_size
required: true
description: >
Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name:
mnist_example
template_name:
mnist_template
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
}{code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
> [Umbrella] Predefined Experiment
>
>
> Key: SUBMARINE-548
> URL: https://
[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169515#comment-17169515
]
Kevin Su edited comment on SUBMARINE-548 at 8/2/20, 11:25 AM:
--
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description:
"Train data is expected in SVM format, and can be stored in HDFS/S3"
name: training.learning_rate
required: true
default: 0.001
description:
"Learning rate for mnist model, default is 0.001"
name: training.batch_size
required: true
description:
"Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32"
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name: "mnist_example"
template_name: "mnist_template"
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
} {code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
was (Author: pingsutw):
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description:
"Train data is expected in SVM format, and can be stored in HDFS/S3"
name: training.learning_rate
required: true
default: 0.001
description:
"Learning rate for mnist model, default is 0.001"
name: training.batch_size
required: true
description:
"Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32"
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name:
mnist_example
template_name:
mnist_template
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
} {code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
> [Umbrella] Predefined Experiment
>
>
> Key: SUBMARINE-548
> URL: https://issues.apache.org/jira/browse/SUBMARINE-548
>
[jira] [Comment Edited] (SUBMARINE-548) [Umbrella] Predefined Experiment
[
https://issues.apache.org/jira/browse/SUBMARINE-548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169515#comment-17169515
]
Kevin Su edited comment on SUBMARINE-548 at 8/2/20, 11:31 AM:
--
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description:
"Train data is expected in SVM format, and can be stored in HDFS/S3"
name: training.learning_rate
required: true
default: 0.001
description:
"Learning rate for mnist model, default is 0.001"
name: training.batch_size
required: true
description:
"Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32"
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name: "mnist_example"
template_name: "mnist_template"
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
} {code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
was (Author: pingsutw):
[~wangda], A few questions, make sure I understand it correctly.
# If I want to use a predefined template to submit an experiment, we would
register the *ExperimentTemplateSpec* first.
*ExperimentTemplateSpec* will look like below
{code:java}
{
template_name: mnist_template
experiment_spec:
meta:
name: name
namespace: "default"
framework: "TensorFlow"
cmd: "python /var/tf_mnist/mnist_with_summaries.py
--input {{input.train_data }}
--log_dir=/train/log
--learning_rate={{training.learning_rate}}
--batch_size={{training.batch_size}}"
envVars:
ENV_1: "ENV1"
environment:
image: "gcr.io/kubeflow-ci/tf-mnist-with-summaries:1.0"
spec:
Ps:
replicas: 1
resources: "cpu=1,memory=1024M"
Worker:
replicas: 1
resources: "cpu=1,memory=1024M"
parameters:
name: input.train_data
required: true
description:
"Train data is expected in SVM format, and can be stored in HDFS/S3"
name: training.learning_rate
required: true
default: 0.001
description:
"Learning rate for mnist model, default is 0.001"
name: training.batch_size
required: true
description:
"Integer or `None`. Number of samples per gradient update. If
unspecified, `batch_size` will default to 32"
} {code}
Should we add *Author* and *description* in *ExperimentTemplateSpec,* as
mention in
[https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experimen|https://github.com/apache/submarine/blob/master/docs/design/experiment-implementation.md#predefined-experiment-template-api-to-run-experiment]
2. After registering, we will submit a list of parameters to run an
experiment like below
{code:java}
{
experiment_name: "mnist_example"
template_name: "mnist_template"
parameters:
input.train_data: "hdsf://foo/bar"
training.learning_rate: 0.01
training.batch_size: 64
} {code}
IIUC, It's a great proposal that users could very easily submit an experiment
with a list of parameters, and no need to worry about other system resources
and the environment.
> [Umbrella] Predefined Experiment
>
>
> Key: SUBMARINE-548
> URL: https://issues.apache.org/jira/browse/SUBMARINE-548
> Project: Apache Submari
