[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[ https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116404#comment-17116404 ] Manikandan R commented on SUBMARINE-507: Closed all sub-tasks as those items will be covered in this JIRA itself. > Submarine Environment Management > > > Key: SUBMARINE-507 > URL: https://issues.apache.org/jira/browse/SUBMARINE-507 > Project: Apache Submarine > Issue Type: New Feature >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Labels: pull-request-available > > Scope of this JIRA is to support environment management. It includes the > following: > 1. Create Environment > 2. Update Environment > 3. Delete Environment > 4. List Environments > In addition, this JIRA should also ensures that environments has been > persisted like experiments so that it can used for later use. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117911#comment-17117911
]
Manikandan R commented on SUBMARINE-507:
Writing down my thoughts on storing environments..Please share your views.
1. Following tables can be created in Submarine Metastore:
a) Table Name: environments
Columns:
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
"location" column captures hdfs path of the environment file.
b) Table Name: docker_images
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
c) Table Name: kernel
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
Having separate tables for docker_images and kernel give us lot of flexibility
while operating environments.
- docker and kernel images could be created only once and used for many
environments.
- Avoid creating the same images and kernel/conda again and again in registry
and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and
becomes a bottleneck, but very unlikely).
2. How to store environment file?
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't
exists and use environment name as file name. For example,
hdfs://mycluster/submarine/environments/my_env.txt
3. How to store docker_images?
We could set up our own registry as part of starting up the server. Please
refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details.
There are several options for this storage as documented in
[https://docs.docker.com/registry/configuration/#storage]. For first cut, We
can begin with file system and can be iterated over next releases based on the
need.
4. How to store kernel/conda?
There are 2 types. 1. Private 2. Public.
For private repo, we will need to set up local repo's and can be used.
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118380#comment-17118380
]
Zhankun Tang commented on SUBMARINE-507:
[~maniraj...@gmail.com], thanks for the update! In my understanding, "A
environment = A base image + A conda env spec". The base Docker images can be
split into two categories:
# Submarine environment base images (OS version, base system libraries,
anaconda installed)
# End-user custom base images (custom OS, system libraries and anaconda)
The conda env spec seems to have anaconda version dependencies.
{{}}
{code:java}
name: "my_submarine_env",
vm-image: "...",
docker-image: "...",
kernel: name: team_default_python_3.7
channels:
- defaults dependencies:
- _ipyw_jlab_nb_ext_conf=0.1.0=py37_0
- alabaster=0.7.12=py37_0
- anaconda=2020.02=py37_0
- anaconda-client=1.7.2=py37_0
- anaconda-navigator=1.9.12=py37_0{code}
{{}}
For base image storage, one simple way, to begin with, might be storing the
submarine base image or release them in Docker hub's "apache/submarine". And
build an entry script to activate the conda spec at runtime.
Later we may set up a custom Docker registry or store the base image in the
user's existing Docker registry.
For end-user custom base image storage, I'd prefer that we can integrate the
end-user's existing Docker registry as a starting.
Thoughts? [~wangda]
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17119816#comment-17119816
]
Wangda Tan commented on SUBMARINE-507:
--
Thanks [~maniraj...@gmail.com], [~ztang] for the comments,
For the environment, I don't think we should get into image storage / Docker
registry setup business, it should just be a Docker image name and K8s can
figure out what credentials to pull the docker image based on secrets stored in
the namespace. And users are responsible for setting up Docker registry and K8s
secrets.
{quote}For base image storage, one simple way, to begin with, might be storing
the submarine base image or release them in Docker hub's "apache/submarine".
And build an entry script to activate the conda spec at runtime.
{quote}
This make sense to me, and this is just an example, user can choose whatever
base Docker image they want to use, it could borrow from the apache/submarine's
base image.
{quote}For end-user custom base image storage, I'd prefer that we can integrate
the end-user's existing Docker registry as a starting.
{quote}
Make sense.
One thing I realized is, if user only want to use a Docker image (instead of
Anaconda), asking users to create an environment, and put Docker image like:
{code:java}
Environment:
name: my-env
docker-image: example.com/my-docker-image:0.1.2
In experiment, user specify
Experiment:
name: "My-tf-job"
environment: "my-env"
task:
script: ...
resource: ...{code}
It is purely an overhead and bad user experiences.
How about we can provide a short cut to specify an "anonymous/embeded
Environment" as part of the Experiment, for example:
{code:java}
Experiment:
name: "My-tf-job"
environment: "my-env" // point to a "named" environment, or
environment:
docker-image: example.com/my-docker-image:0.1.2
kernel-name: ...
task:
script: ...
resource: ... {code}
If it is possible, we should make the "embedded environment" be part of 0.4.0
itself, so we don't have to change this API when we release 0.5.0
Thoughts? [~ztang], [~maniraj...@gmail.com]
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120450#comment-17120450
]
Manikandan R commented on SUBMARINE-507:
Thanks for sharing your views.
IIUC, We would like to have only 1 table "environments" in Submarine Metastore.
Few more questions:
1. How do we decide that we need to talk to Docker hub's "apache/submarine" or
end-user's existing Docker registry? Using docker image name?
2. For both the categories, Can we safely assume images will be always
available for use at run time? Nothing to worry about this while creating
environment.
3. For first category of base images ("apache/submarine"),When & How are we
going to create images and release to "apache/submarine"? Using docker file
manually? Admin would be doing this?
4. For second category of base images, Can we assume end users would have
created the image in their docker registry?
5. It seems we are in favour of storing environment spec as is in a string
based column in "environments" table?
6. conda env activation happens at run time while running the notebook or
experiment. When are we going to create an env in conda?
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120592#comment-17120592
]
Wangda Tan commented on SUBMARINE-507:
--
[~maniraj...@gmail.com],
{quote} # How do we decide that we need to talk to Docker hub's
"apache/submarine" or end-user's existing Docker registry? Using docker image
name?{quote}
Yes, it should based on the image name and Handled by Docker daemon.
{quote}2. For both the categories, Can we safely assume images will be always
available for use at run time? Nothing to worry about this while creating
environment?
{quote}
Yes, if it has any issue (like image cannot be pulled), the experiment run will
fail (naturally).
{quote}3. For first category of base images ("apache/submarine"),When & How are
we going to create images and release to "apache/submarine"? Using docker file
manually? Admin would be doing this?
{quote}
This is part of the Submarine release process, I believe we already doing this,
cc:[~ztang].
{quote}4. For second category of base images, Can we assume end users would
have created the image in their docker registry?
{quote}
Yes, for whatever base image specified, user need to ensure it can be pulled.
{quote}5. It seems we are in favour of storing environment spec as is in a
string based column in "environments" table?
{quote}
It looks like a good solution to me, I'm not sure if there's any other
alternatives.
{quote}6. conda env activation happens at run time while running the notebook
or experiment. When are we going to create an env in conda?
{quote}
Yes, activation only happens at run time. To create an env, I think we can have
following Python APIs:
1) Create Environment using SDK (pseudo code)
{code:java}
env = create_new_environment("my_env")
# Set Docker image
env.set_docker_image("apache/submarine-123:123")
# Set conda kernel using spec
env.add_conda_kernel(conda_kernel_from_spec("""
name: team_default_python_3.7
channels:
- defaults
dependencies:
- _ipyw_jlab_nb_ext_conf=0.1.0=py37_0
- alabaster=0.7.12=py37_0
- anaconda=2020.02=py37_0
- anaconda-client=1.7.2=py37_0
- anaconda-navigator=1.9.12=py37_0
""")
# Alternatively, set conda kernel using APIs
conda_kernel = create_new_conda_kernel("my_kernel")
conda_kernel.set_channels(["..."])
conda_kernel.add_pip_dependency("...")
# Finally, save the env, this will save to metadata
submarine.register_env(env)
{code}
2) Similarily, we need List API, Get API, Delete API for environment.
Thoughts?
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17120732#comment-17120732
]
Zhankun Tang commented on SUBMARINE-507:
[~maniraj...@gmail.com], yeah. We've already published the
"apache/submarine:mini-0.3.0" under "apache/submarine" repo. We can release
more.
[~wangda], for the kernel spec, is the sample spec generated by "condo export"?
It seems hard to write that by hand.
Will the kernel spec has hard dependencies on the conda component versions when
enabling?
For instance, we provide a submarine image with "anaconda=2019.02=py36", but
the kernel spec is "2020.02=py37". Will the image fail to start due to version
mismatch?
In other words. Is the kernel spec portability a concern for us?
{code:java}
- anaconda=2020.02=py37_0
- anaconda-client=1.7.2=py37_0
- anaconda-navigator=1.9.12=py37_0
{code}
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17121444#comment-17121444
]
Wangda Tan commented on SUBMARINE-507:
--
[~ztang], good point, here's my thoughts:
{quote}Will the kernel spec has hard dependencies on the conda component
versions when enabling?
{quote}
Based on the Anaconda's typical usage, version of Python is part of the kernel,
so in that case. Python inside the image will be ignored. (Anaconda uses
environment variables to only pass whatever libraries when the kernel is
activated).
{quote}[~wangda], for the kernel spec, is the sample spec generated by "condo
export"? It seems hard to write that by hand.
{quote}
That's correct, ideally user should not write the spec. Once a kernel is
activated, all the activities like {{pip install}} will be tracked and when
user call {{conda export}} such changes will be part of the exported conda spec.
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
