[jira] [Commented] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117997#comment-17117997 ] Wangda Tan commented on SUBMARINE-519: -- Marked it to be 0.4.0 blocker so we can get docs fixed before 0.4.0 release. > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Target Version: 0.4.0 > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Priority: Blocker (was: Major) > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Updated] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated SUBMARINE-519: - Description: Targets: 1) Cleanup obsolete contents. 2) Update vision to the latest vision. 3) Make a clear boundary between the developer doc and user doc. > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > > Targets: > 1) Cleanup obsolete contents. > 2) Update vision to the latest vision. > 3) Make a clear boundary between the developer doc and user doc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Assigned] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
[ https://issues.apache.org/jira/browse/SUBMARINE-519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned SUBMARINE-519: Assignee: Wangda Tan > Overhaul Structure of Submarine Github Markdown Files > - > > Key: SUBMARINE-519 > URL: https://issues.apache.org/jira/browse/SUBMARINE-519 > Project: Apache Submarine > Issue Type: Sub-task > Components: Doc >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-519) Overhaul Structure of Submarine Github Markdown Files
Wangda Tan created SUBMARINE-519: Summary: Overhaul Structure of Submarine Github Markdown Files Key: SUBMARINE-519 URL: https://issues.apache.org/jira/browse/SUBMARINE-519 Project: Apache Submarine Issue Type: Sub-task Components: Doc Reporter: Wangda Tan -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
[ https://issues.apache.org/jira/browse/SUBMARINE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117996#comment-17117996 ] Wangda Tan commented on SUBMARINE-518: -- We also want to make use of tools like Sphinx to generate static html from source control file, it will be especially useful for Python-related APIs, etc. For example: [https://pytorch.org/docs/stable/index.html] As we're doing more Python SDK API, we also need to get this documentation versioned since we will have different APIs between versions, and tools like Sphinx can generate API doc from source file, which help a lot of efforts. Ideally, documentation can be built nightly so there's always a doc point to the master branch (Like pytorch example mentioned above). In the short term, it is fine to make documentation part of Github, but we need to make a clear structure between these markdown files. > [Umbrella] Overhaul Submarine Documentation And Website > --- > > Key: SUBMARINE-518 > URL: https://issues.apache.org/jira/browse/SUBMARINE-518 > Project: Apache Submarine > Issue Type: Task >Reporter: Wangda Tan >Priority: Major > > Now we have a number of issues of our Documentation and Website: > 1) There's no enough distinguish between website / Github documentation. > 2) There's no site to host documentation based on versions. > We need find a way to solve this problem, that's the main purpose of the > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Commented] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
[ https://issues.apache.org/jira/browse/SUBMARINE-518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117994#comment-17117994 ] Wangda Tan commented on SUBMARINE-518: -- Here's the proposal: Make clear responsibilities between website and github docs: * Website is responsible for: ** Showing the vision of the project. ** Provide links to *User-Documentation(How to use, how to install, how to build)*/Download. ** Provide links to Github, Blogpost. * Roadmap is documented on WIKI: ** [https://cwiki.apache.org/confluence/display/SUBMARINE/Roadmap] ** The reason to put roadmap on Atlassian WIKI is, it is frequently changing, we want release managers can easier modify roadmaps. ** Also, integration from confluent to Jira is helpful to track the release progress. * Github is responsible for: ** Source control of following contents: *** Landing page of Github (README.md) should include following contents. (Short) Description of the project Link to the User Documentation. *** CONTRIBUTOR.md > [Umbrella] Overhaul Submarine Documentation And Website > --- > > Key: SUBMARINE-518 > URL: https://issues.apache.org/jira/browse/SUBMARINE-518 > Project: Apache Submarine > Issue Type: Task >Reporter: Wangda Tan >Priority: Major > > Now we have a number of issues of our Documentation and Website: > 1) There's no enough distinguish between website / Github documentation. > 2) There's no site to host documentation based on versions. > We need find a way to solve this problem, that's the main purpose of the > Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Created] (SUBMARINE-518) [Umbrella] Overhaul Submarine Documentation And Website
Wangda Tan created SUBMARINE-518: Summary: [Umbrella] Overhaul Submarine Documentation And Website Key: SUBMARINE-518 URL: https://issues.apache.org/jira/browse/SUBMARINE-518 Project: Apache Submarine Issue Type: Task Reporter: Wangda Tan Now we have a number of issues of our Documentation and Website: 1) There's no enough distinguish between website / Github documentation. 2) There's no site to host documentation based on versions. We need find a way to solve this problem, that's the main purpose of the Jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org
[jira] [Comment Edited] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117911#comment-17117911
]
Manikandan R edited comment on SUBMARINE-507 at 5/27/20, 4:47 PM:
--
Writing down my thoughts on storing environments..Please share your views.
1. Following tables can be created in Submarine Metastore:
a) Table Name: environments
Columns:
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
"location" column captures hdfs path of the environment file.
b) Table Name: docker_images
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
c) Table Name: kernel
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
Having separate tables for docker_images and kernel give us lot of flexibility
while operating environments. a) docker and kernel images could be created only
once and used for many environments. b) Avoid creating the same images and
kernel/conda again and again in registry and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and
becomes a bottleneck, but very unlikely).
2. How to store environment file?
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't
exists and use environment name as file name. For example,
hdfs://mycluster/submarine/environments/my_env.txt
We could possibly provide more options like HDFS in future phases.
3. How to store docker_images?
We could set up our own registry as part of starting up the server. Please
refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details.
There are several options for this storage as documented in
[https://docs.docker.com/registry/configuration/#storage]. For first cut, We
can begin with file system and can be iterated over next releases based on the
need.
4. How to store kernel/conda?
There are 2 types. 1. Private 2. Public.
For private repo, we will need to set up local repo's and can be used.
was (Author: maniraj...@gmail.com):
Writing down my thoughts on storing environments..Please share your views.
1. Following tables can be created in Submarine Metastore:
a) Table Name: environments
Columns:
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
"location" column captures hdfs path of the environment file.
b) Table Name: docker_images
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
c) Table Name: kernel
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
Having separate tables for docker_images and kernel give us lot of flexibility
while operating environments.
- docker and kernel images could be created only once and used for many
environments.
- Avoid creating the same images and kernel/conda again and again in registry
and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and
becomes a bottleneck, but very unlikely).
2. How to store environment file?
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't
exists and use environment name as file name. For example,
hdfs://mycluster/submarine/environments/my_env.txt
3. How to store docker_images?
We could set up our own registry as part of starting up the server. Please
refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details.
There are several options for this storage as documented in
[https://docs.docker.com/registry/configuration/#storage]. For first cut, We
can begin with file system and can be iterated over next releases based on the
need.
4. How to store kernel/conda?
There are 2 types. 1. Private 2. Public.
For private repo, we will need to set up local repo's and can be used.
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
>
[jira] [Commented] (SUBMARINE-507) Submarine Environment Management
[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117911#comment-17117911
]
Manikandan R commented on SUBMARINE-507:
Writing down my thoughts on storing environments..Please share your views.
1. Following tables can be created in Submarine Metastore:
a) Table Name: environments
Columns:
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
"location" column captures hdfs path of the environment file.
b) Table Name: docker_images
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
c) Table Name: kernel
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
Having separate tables for docker_images and kernel give us lot of flexibility
while operating environments.
- docker and kernel images could be created only once and used for many
environments.
- Avoid creating the same images and kernel/conda again and again in registry
and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and
becomes a bottleneck, but very unlikely).
2. How to store environment file?
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't
exists and use environment name as file name. For example,
hdfs://mycluster/submarine/environments/my_env.txt
3. How to store docker_images?
We could set up our own registry as part of starting up the server. Please
refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details.
There are several options for this storage as documented in
[https://docs.docker.com/registry/configuration/#storage]. For first cut, We
can begin with file system and can be iterated over next releases based on the
need.
4. How to store kernel/conda?
There are 2 types. 1. Private 2. Public.
For private repo, we will need to set up local repo's and can be used.
> Submarine Environment Management
>
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org
For additional commands, e-mail: dev-h...@submarine.apache.org
