tangzhankun commented on a change in pull request #61: SUBMARINE-189. [SS] Add submarine server architecture doc URL: https://github.com/apache/hadoop-submarine/pull/61#discussion_r338884001
########## File path: docs/design/submarine-server/architecture.md ########## @@ -0,0 +1,126 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. + --> +## Motivation +Due to the single responsibility of considering different components, and nowadays the core module depends the hadoop libs which can be setted optional. + +## Proposal + ``` + +---------------------+ + +-----------+ | +--------+ +----+ | + | | | |runtime1+-->+job1| | + | workbench +---+ +---------------------------------+ | +--------+ +----+ | + | | | | +------+ +--------------------+ | +-->+ +--------+ +----+ | + +-----------+ | | | | | +------+ +-------+ | | | | |runtime2+-->+job2| | + | | | | | | YARN | | K8s | | | | | +--------+ +----+ | + +-----------+ | | | | | +------+ +-------+ | | | | YARN Cluster | + | | | | | | | submitter | | | +---------------------+ + | CLI +------->+ | | +--------------------+ +---+ + | | | | | REST | +--------+ +---------+ | | +---------------------+ + +-----------+ | | | | | TRANSL | | monitor | | | | +--------+ +----+ | + | | | | +--------+ +---------+ | | | | +-->+job1| | + +-----------+ | | | | +--------------------+ | | | | | +----+ | + | | | | | | | JobManager | | +-->+ |operator| +----+ | + | SDK +---+ | +------+ +--------------------+ | | | +-->+job2| | + | | +---------------------------------+ | +--------+ +----+ | + +-----------+ | K8s Cluster | + client server +---------------------+ + ``` +We propose to split the core module to two modules, CLI and server as shown in FIG. The client call the REST API to submit and retrieve the job info. + +### Submarine module structure +``` + |--submarine + |--submarine-all + |--submarine-client + |--submarine-commons + |--submarine-dist + |--submarine-sdk + | |--pysubmarine + |--submarine-server + | |--server-core + | |--server-submitter + | | |----submitter-yarn + | | |----submitter-yarnservice + | | |----submitter-k8s + | |--server-yarn-runtime + | |--server-operator + |--submarine-workbench + | |--workbench-server + | |--workbench-web + |--submodules +``` + +### submarine-client +The client as the default implements for submarine server RESTful API, it provides all the features which declared in the RESTful API. + +### submarine-server +``` ++------+ +--------------------+ +| | | +------+ +-------+ | +| | | | YARN | | K8s | | +| | | +------+ +-------+ | +| | | submitter | +| | +--------------------+ +| REST | +--------+ +---------+ +| | | TRANSL | | monitor | +| | +--------+ +---------+ +| | +--------------------+ +| | | JobManager | ++------+ +--------------------+ +``` +The project structure is as follows: +``` + |--submarine-server + |----server-api + |----server-core + |----server-submitter + |----pom.xml + |----README +``` +The server-sdk will be added in the future. It is the official client for Submarine Server API. + +#### REST +It provides the RESTful API for users, consists of submit job and get job info features and so on. Often used by CLI and workbench etc. The submarine client project should implement all APIs. + +#### Submitter +It provides the submitter interface, and implement it for YARN and K8s. Besides that support extends the other resource management system such as submarine-docker-cluster only implement the interface which declared at server-core project. So that support submit the machine learning job to different cluster resource management system. The project structure is as follows: +``` + |--server-submitter + |----submitter-yarn + |----submitter-yarnservice + |----submitter-k8s + |----pom.xml + |----README +``` +In here the submitter-yarn implement the TonY Runtime(tony-runtime), and the submitter-yarnservice(yarnservice-runtime) diabled by default until we decide to support the yarn service feature. If we want to support the submarine-docker-cluster (Submarine uses the Raft algorithm to create a docker cluster on the docker runtime environment on multiple servers, providing the most lightweight resource scheduling system for small-scale users.) should create the sub project named submitter-docker and must implement all the interface declared in the server-core project. + +#### Monitor +We need a monitor to track the job life cycle and record the main events and key info in running. If the client register to server, the monitor will called the pusher to push the info to client. + +#### Translator Review comment: I'm wondering if this translator logic should be inside each submitter. Thoughts? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
