This is an automated email from the ASF dual-hosted git repository. pingsutw pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push: new 4b30e71 SUBMARINE-953. Submarine-server shouldn't create persistentVolume manually for notebook 4b30e71 is described below commit 4b30e71ec803e3c8209a73477fdea54d8a463869 Author: Kenchu123 <k889...@gmail.com> AuthorDate: Sat Aug 7 18:53:51 2021 +0800 SUBMARINE-953. Submarine-server shouldn't create persistentVolume manually for notebook ### What is this PR for? <!-- A few sentences describing the overall goals of the pull request's commits. First time? Check out the contributing guide - https://submarine.apache.org/contribution/contributions.html --> Source: https://github.com/apache/submarine/blob/72be805fc4672f7f29c96cca95e16d59358d4dd2/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java#L379 We can find out that when we create a notebook custom resource, the server will manually create the persistent volume for it. However, this made submarine-server has to grant access to persistent volume resources, which lead to setting cluster roles for the server. (https://github.com/apache/submarine/blob/72be805fc4/helm-charts/submarine/templates/rbac.yaml#L61) Since the server is in namespace scope, giving it cluster roles is inappropriate. Besides, this makes multi-tenancy more difficult. To fix the bug, we can create a storage class to dynamically provision persistent volumes, and submarine-server will only need to deal with persistent volume claim. Ref: - storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/#local - notebook-cr spec: https://www.kubeflow.org/docs/reference/notebook/v1/ ### What type of PR is it? [Bug Fix] ### Todos * [x] - add storageClass * [x] - add persistentVolumeClaim spec to submarine-server * [x] - remove submarine-server createPersistentVolume * [x] - replace clutserrolebinding to rolebinding in helm-charts * [x] - replace clusterrole to role in helm-charts * [x] - modify submarine operator ### What is the Jira issue? <!-- * Open an issue on Jira https://issues.apache.org/jira/browse/SUBMARINE/ * Put link here, and add [SUBMARINE-*Jira number*] in PR title, eg. `SUBMARINE-23. PR title` --> https://issues.apache.org/jira/browse/SUBMARINE-953 ### How should this be tested? <!-- * First time? Setup Travis CI as described on https://submarine.apache.org/contribution/contributions.html#continuous-integration * Strongly recommended: add automated unit tests for any new or changed behavior * Outline any manual steps to test the PR here. --> Can be tested either using `helm install` or `submarine-operator`. ### Screenshots (if appropriate) https://user-images.githubusercontent.com/17617373/128290829-ff38828c-3555-4c0c-b37d-6b147b762dde.mov ### Questions: * Do the license files need updating? No * Are there breaking changes for older versions? No * Does this need new documentation? No Author: Kenchu123 <k889...@gmail.com> Signed-off-by: Kevin <pings...@apache.org> Closes #697 from Kenchu123/SUBMARINE-953 and squashes the following commits: c93d7124 [Kenchu123] SUBMARINE-953. Add Apache License to helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml 877f8f2b [Kenchu123] SUBMARINE-953. Remove clusterrole and clusterrolebinding in controller 6a1b8f1d [Kenchu123] SUBMARINE-953. Fix go format for controller.go a5ad7430 [Kenchu123] SUBMARINE-953. remove quotation mark in helm-charts notebook-controller storageclass 36713137 [Kenchu123] SUBMARINE-953. Change ClusterRole to Role in submarine controller. Add storageClass to helm charts. 267c0714 [Kenchu123] SUBMARINE-953. Create Notebook with pvc and storageClass, and remove pv creating and chage clusterrole to role --- .../templates/storageclass.yaml | 24 ++++ helm-charts/submarine/templates/rbac.yaml | 7 +- .../submarine/templates/submarine-server.yaml | 3 +- submarine-cloud-v2/main.go | 4 +- submarine-cloud-v2/pkg/controller/controller.go | 28 ++--- .../pkg/controller/submarine_server_rbac.go | 38 +++--- .../server/submitter/k8s/K8sSubmitter.java | 138 +++++++-------------- .../submitter/k8s/parser/NotebookSpecParser.java | 10 +- .../submitter/k8s/parser/VolumeSpecParser.java | 7 +- .../server/submitter/k8s/util/NotebookUtils.java | 2 + 10 files changed, 124 insertions(+), 137 deletions(-) diff --git a/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml b/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml new file mode 100644 index 0000000..de8a3d8 --- /dev/null +++ b/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml @@ -0,0 +1,24 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +apiVersion: storage.k8s.io/v1 +kind: StorageClass +metadata: + name: notebook-storageclass + labels: + app: notebook-controller +provisioner: k8s.io/minikube-hostpath +reclaimPolicy: Delete diff --git a/helm-charts/submarine/templates/rbac.yaml b/helm-charts/submarine/templates/rbac.yaml index ff77fdf..fce2ef6 100644 --- a/helm-charts/submarine/templates/rbac.yaml +++ b/helm-charts/submarine/templates/rbac.yaml @@ -16,7 +16,7 @@ # apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole +kind: Role metadata: name: "{{ .Values.submarine.server.name }}" rules: @@ -58,7 +58,6 @@ rules: - pods - pods/log - services - - persistentvolumes - persistentvolumeclaims verbs: - '*' @@ -70,7 +69,7 @@ rules: verbs: - '*' --- -kind: ClusterRoleBinding +kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: "{{ .Values.submarine.server.name }}" @@ -79,6 +78,6 @@ subjects: namespace: {{ .Release.Namespace }} name: "{{ .Values.submarine.server.name }}" roleRef: - kind: ClusterRole + kind: Role name: "{{ .Values.submarine.server.name }}" apiGroup: rbac.authorization.k8s.io diff --git a/helm-charts/submarine/templates/submarine-server.yaml b/helm-charts/submarine/templates/submarine-server.yaml index 2452115..5a3f063 100644 --- a/helm-charts/submarine/templates/submarine-server.yaml +++ b/helm-charts/submarine/templates/submarine-server.yaml @@ -69,5 +69,4 @@ spec: image: "{{ .Values.submarine.server.image }}" imagePullPolicy: {{ .Values.submarine.server.imagePullPolicy }} ports: - - containerPort: 8080 - + - containerPort: 8080 \ No newline at end of file diff --git a/submarine-cloud-v2/main.go b/submarine-cloud-v2/main.go index 8bdf512..9db241c 100644 --- a/submarine-cloud-v2/main.go +++ b/submarine-cloud-v2/main.go @@ -95,8 +95,8 @@ func main() { kubeInformerFactory.Core().V1().PersistentVolumeClaims(), kubeInformerFactory.Extensions().V1beta1().Ingresses(), traefikInformerFactory.Traefik().V1alpha1().IngressRoutes(), - kubeInformerFactory.Rbac().V1().ClusterRoles(), - kubeInformerFactory.Rbac().V1().ClusterRoleBindings(), + kubeInformerFactory.Rbac().V1().Roles(), + kubeInformerFactory.Rbac().V1().RoleBindings(), submarineInformerFactory.Submarine().V1alpha1().Submarines()) // notice that there is no need to run Start methods in a separate goroutine. (i.e. go kubeInformerFactory.Start(stopCh) diff --git a/submarine-cloud-v2/pkg/controller/controller.go b/submarine-cloud-v2/pkg/controller/controller.go index 2c0ad80..d4af685 100644 --- a/submarine-cloud-v2/pkg/controller/controller.go +++ b/submarine-cloud-v2/pkg/controller/controller.go @@ -124,8 +124,8 @@ type Controller struct { persistentvolumeclaimLister corelisters.PersistentVolumeClaimLister ingressLister extlisters.IngressLister ingressrouteLister traefiklisters.IngressRouteLister - clusterroleLister rbaclisters.ClusterRoleLister - clusterrolebindingLister rbaclisters.ClusterRoleBindingLister + roleLister rbaclisters.RoleLister + rolebindingLister rbaclisters.RoleBindingLister // workqueue is a rate limited work queue. This is used to queue work to be // processed instead of performing it as soon as a change happens. This // means we can ensure we only process a fixed amount of resources at a @@ -153,8 +153,8 @@ func NewController( persistentvolumeclaimInformer coreinformers.PersistentVolumeClaimInformer, ingressInformer extinformers.IngressInformer, ingressrouteInformer traefikinformers.IngressRouteInformer, - clusterroleInformer rbacinformers.ClusterRoleInformer, - clusterrolebindingInformer rbacinformers.ClusterRoleBindingInformer, + roleInformer rbacinformers.RoleInformer, + rolebindingInformer rbacinformers.RoleBindingInformer, submarineInformer informers.SubmarineInformer) *Controller { // Add Submarine types to the default Kubernetes Scheme so Events can be @@ -181,8 +181,8 @@ func NewController( persistentvolumeclaimLister: persistentvolumeclaimInformer.Lister(), ingressLister: ingressInformer.Lister(), ingressrouteLister: ingressrouteInformer.Lister(), - clusterroleLister: clusterroleInformer.Lister(), - clusterrolebindingLister: clusterrolebindingInformer.Lister(), + roleLister: roleInformer.Lister(), + rolebindingLister: rolebindingInformer.Lister(), workqueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "Submarines"), recorder: recorder, incluster: incluster, @@ -294,24 +294,24 @@ func NewController( }, DeleteFunc: controller.handleObject, }) - clusterroleInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ + roleInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: controller.handleObject, UpdateFunc: func(old, new interface{}) { - newClusterRole := new.(*rbacv1.ClusterRole) - oldClusterRole := old.(*rbacv1.ClusterRole) - if newClusterRole.ResourceVersion == oldClusterRole.ResourceVersion { + newRole := new.(*rbacv1.Role) + oldRole := old.(*rbacv1.Role) + if newRole.ResourceVersion == oldRole.ResourceVersion { return } controller.handleObject(new) }, DeleteFunc: controller.handleObject, }) - clusterrolebindingInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ + rolebindingInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{ AddFunc: controller.handleObject, UpdateFunc: func(old, new interface{}) { - newClusterRoleBinding := new.(*rbacv1.ClusterRoleBinding) - oldClusterRoleBinding := old.(*rbacv1.ClusterRoleBinding) - if newClusterRoleBinding.ResourceVersion == oldClusterRoleBinding.ResourceVersion { + newRoleBinding := new.(*rbacv1.RoleBinding) + oldRoleBinding := old.(*rbacv1.RoleBinding) + if newRoleBinding.ResourceVersion == oldRoleBinding.ResourceVersion { return } controller.handleObject(new) diff --git a/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go b/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go index 431b30c..2b92b1e 100644 --- a/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go +++ b/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go @@ -30,8 +30,8 @@ import ( "k8s.io/klog/v2" ) -func newSubmarineServerClusterRole(submarine *v1alpha1.Submarine) *rbacv1.ClusterRole { - return &rbacv1.ClusterRole{ +func newSubmarineServerRole(submarine *v1alpha1.Submarine) *rbacv1.Role { + return &rbacv1.Role{ ObjectMeta: metav1.ObjectMeta{ Name: serverName, OwnerReferences: []metav1.OwnerReference{ @@ -63,8 +63,8 @@ func newSubmarineServerClusterRole(submarine *v1alpha1.Submarine) *rbacv1.Cluste } } -func newSubmarineServerClusterRoleBinding(submarine *v1alpha1.Submarine) *rbacv1.ClusterRoleBinding { - return &rbacv1.ClusterRoleBinding{ +func newSubmarineServerRoleBinding(submarine *v1alpha1.Submarine) *rbacv1.RoleBinding { + return &rbacv1.RoleBinding{ ObjectMeta: metav1.ObjectMeta{ Name: serverName, OwnerReferences: []metav1.OwnerReference{ @@ -79,7 +79,7 @@ func newSubmarineServerClusterRoleBinding(submarine *v1alpha1.Submarine) *rbacv1 }, }, RoleRef: rbacv1.RoleRef{ - Kind: "ClusterRole", + Kind: "Role", Name: serverName, APIGroup: "rbac.authorization.k8s.io", }, @@ -91,12 +91,12 @@ func newSubmarineServerClusterRoleBinding(submarine *v1alpha1.Submarine) *rbacv1 func (c *Controller) createSubmarineServerRBAC(submarine *v1alpha1.Submarine) error { klog.Info("[createSubmarineServerRBAC]") - // Step1: Create ClusterRole - clusterrole, err := c.clusterroleLister.Get(serverName) + // Step1: Create Role + role, err := c.roleLister.Roles(submarine.Namespace).Get(serverName) // If the resource doesn't exist, we'll create it if errors.IsNotFound(err) { - clusterrole, err = c.kubeclientset.RbacV1().ClusterRoles().Create(context.TODO(), newSubmarineServerClusterRole(submarine), metav1.CreateOptions{}) - klog.Info(" Create ClusterRole: ", clusterrole.Name) + role, err = c.kubeclientset.RbacV1().Roles(submarine.Namespace).Create(context.TODO(), newSubmarineServerRole(submarine), metav1.CreateOptions{}) + klog.Info(" Create Role: ", role.Name) } // If an error occurs during Get/Create, we'll requeue the item so we can @@ -106,28 +106,28 @@ func (c *Controller) createSubmarineServerRBAC(submarine *v1alpha1.Submarine) er return err } - if !metav1.IsControlledBy(clusterrole, submarine) { - msg := fmt.Sprintf(MessageResourceExists, clusterrole.Name) + if !metav1.IsControlledBy(role, submarine) { + msg := fmt.Sprintf(MessageResourceExists, role.Name) c.recorder.Event(submarine, corev1.EventTypeWarning, ErrResourceExists, msg) return fmt.Errorf(msg) } - clusterrolebinding, clusterrolebinding_err := c.clusterrolebindingLister.Get(serverName) + rolebinding, rolebinding_err := c.rolebindingLister.RoleBindings(submarine.Namespace).Get(serverName) // If the resource doesn't exist, we'll create it - if errors.IsNotFound(clusterrolebinding_err) { - clusterrolebinding, clusterrolebinding_err = c.kubeclientset.RbacV1().ClusterRoleBindings().Create(context.TODO(), newSubmarineServerClusterRoleBinding(submarine), metav1.CreateOptions{}) - klog.Info(" Create ClusterRoleBinding: ", clusterrolebinding.Name) + if errors.IsNotFound(rolebinding_err) { + rolebinding, rolebinding_err = c.kubeclientset.RbacV1().RoleBindings(submarine.Namespace).Create(context.TODO(), newSubmarineServerRoleBinding(submarine), metav1.CreateOptions{}) + klog.Info(" Create RoleBinding: ", rolebinding.Name) } // If an error occurs during Get/Create, we'll requeue the item so we can // attempt processing again later. This could have been caused by a // temporary network failure, or any other transient reason. - if clusterrolebinding_err != nil { - return clusterrolebinding_err + if rolebinding_err != nil { + return rolebinding_err } - if !metav1.IsControlledBy(clusterrolebinding, submarine) { - msg := fmt.Sprintf(MessageResourceExists, clusterrolebinding.Name) + if !metav1.IsControlledBy(rolebinding, submarine) { + msg := fmt.Sprintf(MessageResourceExists, rolebinding.Name) c.recorder.Event(submarine, corev1.EventTypeWarning, ErrResourceExists, msg) return fmt.Errorf(msg) } diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java index 23a862d..900419f 100644 --- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java +++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java @@ -39,9 +39,7 @@ import io.kubernetes.client.apis.CustomObjectsApi; import io.kubernetes.client.models.V1DeleteOptionsBuilder; import io.kubernetes.client.models.V1Deployment; import io.kubernetes.client.models.V1ObjectMeta; -import io.kubernetes.client.models.V1PersistentVolume; import io.kubernetes.client.models.V1PersistentVolumeClaim; -import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource; import io.kubernetes.client.models.V1Pod; import io.kubernetes.client.models.V1PodList; import io.kubernetes.client.models.V1Service; @@ -359,70 +357,67 @@ public class K8sSubmitter implements Submitter { public Notebook createNotebook(NotebookSpec spec) throws SubmarineRuntimeException { Notebook notebook; final String name = spec.getMeta().getName(); - final String pvName = NotebookUtils.PV_PREFIX + name; + final String scName = NotebookUtils.SC_NAME; final String host = NotebookUtils.HOST_PATH; final String storage = NotebookUtils.STORAGE; final String pvcName = NotebookUtils.PVC_PREFIX + name; String namespace = "default"; - NotebookCR notebookCR; - + if (System.getenv(ENV_NAMESPACE) != null) { namespace = System.getenv(ENV_NAMESPACE); } - + + // parse notebook custom resource + NotebookCR notebookCR; try { - // create notebook custom resource notebookCR = NotebookSpecParser.parseNotebook(spec); Map<String, String> labels = new HashMap<>(); labels.put(NotebookCR.NOTEBOOK_OWNER_SELECTOR_KET, spec.getMeta().getOwnerId()); notebookCR.getMetadata().setLabels(labels); notebookCR.getMetadata().setNamespace(namespace); - - // create persistent volume - createPersistentVolume(pvName, host, storage); - } catch (ApiException e) { - LOG.error("K8s submitter: Create persistent volume for Notebook object failed by " + e.getMessage(), e); - throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: Create persistent volume for " + - "Notebook object failed by " + e.getMessage()); } catch (JsonSyntaxException e) { LOG.error("K8s submitter: parse response object failed by " + e.getMessage(), e); throw new SubmarineRuntimeException(500, "K8s Submitter parse upstream response failed."); } - + // create persistent volume claim try { - createPersistentVolumeClaim(pvcName, namespace, pvName, storage); + createPersistentVolumeClaim(pvcName, namespace, scName, storage); } catch (ApiException e) { LOG.error("K8s submitter: Create persistent volume claim for Notebook object failed by " + e.getMessage(), e); - rollbackCreationPV(pvName); throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: Create persistent volume claim for " + "Notebook object failed by " + e.getMessage()); } - - // bind persistent volume claim + + // create notebook custom resource try { - V1PersistentVolumeClaimVolumeSource pvcSource = new V1PersistentVolumeClaimVolumeSource() - .claimName(pvcName); - notebookCR.getSpec().getTemplate().getSpec().getVolumes().get(0).persistentVolumeClaim(pvcSource); - Object object = api.createNamespacedCustomObject(notebookCR.getGroup(), notebookCR.getVersion(), namespace, notebookCR.getPlural(), notebookCR, "true"); notebook = NotebookUtils.parseObject(object, NotebookUtils.ParseOpt.PARSE_OPT_CREATE); - - // create Traefik custom resource - createIngressRoute(notebookCR.getMetadata().getNamespace(), notebookCR.getMetadata().getName()); - } catch (JsonSyntaxException e) { LOG.error("K8s submitter: parse response object failed by " + e.getMessage(), e); - rollbackCreationPVC(pvName, pvcName, namespace); + rollbackCreationPVC(pvcName, namespace); throw new SubmarineRuntimeException(500, "K8s Submitter parse upstream response failed."); } catch (ApiException e) { LOG.error("K8s submitter: parse Notebook object failed by " + e.getMessage(), e); - rollbackCreationPVC(pvName, pvcName, namespace); + rollbackCreationPVC(pvcName, namespace); throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: parse Notebook object failed by " + e.getMessage()); } + + // create notebook Traefik custom resource + try { + createIngressRoute(notebookCR.getMetadata().getNamespace(), notebookCR.getMetadata().getName()); + } catch (ApiException e) { + LOG.error("K8s submitter: Create ingressroute for Notebook object failed by " + + e.getMessage(), e); + rollbackCreationNotebook(notebookCR, namespace); + rollbackCreationPVC(pvcName, namespace); + throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: ingressroute for Notebook " + + "object failed by " + e.getMessage()); + } + return notebook; } @@ -451,7 +446,6 @@ public class K8sSubmitter implements Submitter { public Notebook deleteNotebook(NotebookSpec spec) throws SubmarineRuntimeException { Notebook notebook; final String name = spec.getMeta().getName(); - final String pvName = NotebookUtils.PV_PREFIX + name; final String pvcName = NotebookUtils.PVC_PREFIX + name; String namespace = "default"; @@ -469,7 +463,6 @@ public class K8sSubmitter implements Submitter { notebook = NotebookUtils.parseObject(object, NotebookUtils.ParseOpt.PARSE_OPT_DELETE); deleteIngressRoute(namespace, notebookCR.getMetadata().getName()); deletePersistentVolumeClaim(pvcName, namespace); - deletePersistentVolume(pvName); } catch (ApiException e) { throw new SubmarineRuntimeException(e.getCode(), e.getMessage()); } @@ -479,9 +472,16 @@ public class K8sSubmitter implements Submitter { @Override public List<Notebook> listNotebook(String id) throws SubmarineRuntimeException { List<Notebook> notebookList; + + String namespace = "default"; + + if (System.getenv(ENV_NAMESPACE) != null) { + namespace = System.getenv(ENV_NAMESPACE); + } + try { - Object object = api.listClusterCustomObject(NotebookCR.CRD_NOTEBOOK_GROUP_V1, - NotebookCR.CRD_NOTEBOOK_VERSION_V1, NotebookCR.CRD_NOTEBOOK_PLURAL_V1, + Object object = api.listNamespacedCustomObject(NotebookCR.CRD_NOTEBOOK_GROUP_V1, + NotebookCR.CRD_NOTEBOOK_VERSION_V1, namespace , NotebookCR.CRD_NOTEBOOK_PLURAL_V1, "true", null, NotebookCR.NOTEBOOK_OWNER_SELECTOR_KET + "=" + id, null, null, null); notebookList = NotebookUtils.parseObjectForList(object); @@ -557,49 +557,9 @@ public class K8sSubmitter implements Submitter { } } - public void createPersistentVolume(String pvName, String hostPath, String storage) throws ApiException { - V1PersistentVolume pv = VolumeSpecParser.parsePersistentVolume(pvName, hostPath, storage); - - try { - V1PersistentVolume result = coreApi.createPersistentVolume(pv, "true", null, null); - } catch (ApiException e) { - LOG.error("Exception when creating persistent volume " + e.getMessage(), e); - throw e; - } - } - - public void deletePersistentVolume(String pvName) throws ApiException { - /* - This version of Kubernetes-client/java has bug here. - It will trigger exception as in https://github.com/kubernetes-client/java/issues/86 - but it can still work fine and delete the PV. - */ - try { - V1Status result = coreApi.deletePersistentVolume( - pvName, "true", null, - null, null, null, null - ); - } catch (ApiException e) { - LOG.error("Exception when deleting persistent volume " + e.getMessage(), e); - throw e; - } catch (JsonSyntaxException e) { - if (e.getCause() instanceof IllegalStateException) { - IllegalStateException ise = (IllegalStateException) e.getCause(); - if (ise.getMessage() != null && ise.getMessage().contains("Expected a string but was BEGIN_OBJECT")) { - LOG.debug("Catching exception because of issue " + - "https://github.com/kubernetes-client/java/issues/86", e); - } else { - throw e; - } - } else { - throw e; - } - } - } - - public void createPersistentVolumeClaim(String pvcName, String namespace, String volume, String storage) + public void createPersistentVolumeClaim(String pvcName, String namespace, String scName, String storage) throws ApiException { - V1PersistentVolumeClaim pvc = VolumeSpecParser.parsePersistentVolumeClaim(pvcName, volume, storage); + V1PersistentVolumeClaim pvc = VolumeSpecParser.parsePersistentVolumeClaim(pvcName, scName, storage); try { V1PersistentVolumeClaim result = coreApi.createNamespacedPersistentVolumeClaim( @@ -651,7 +611,7 @@ public class K8sSubmitter implements Submitter { } } - private void createIngressRoute(String namespace, String name) { + private void createIngressRoute(String namespace, String name) throws ApiException { try { IngressRoute ingressRoute = new IngressRoute(); V1ObjectMeta meta = new V1ObjectMeta(); @@ -706,30 +666,28 @@ public class K8sSubmitter implements Submitter { return spec; } - private void rollbackCreationPV(String pvName) { - try { - deletePersistentVolume(pvName); - } catch (ApiException e) { - LOG.error("K8s submitter: delete persistent volume failed by {}, may cause some dirty data", - e.getMessage()); - } - } - - private void rollbackCreationPVC(String pvName, String pvcName, String namespace) { + private void rollbackCreationPVC(String pvcName, String namespace) { try { deletePersistentVolumeClaim(pvcName, namespace); } catch (ApiException e) { LOG.error("K8s submitter: delete persistent volume claim failed by {}, may cause some dirty data", e.getMessage()); } + } + + private void rollbackCreationNotebook(NotebookCR notebookCR, String namespace) + throws SubmarineRuntimeException { try { - deletePersistentVolume(pvName); + Object object = api.deleteNamespacedCustomObject(notebookCR.getGroup(), notebookCR.getVersion(), + namespace, notebookCR.getPlural(), + notebookCR.getMetadata().getName(), + new V1DeleteOptionsBuilder().withApiVersion(notebookCR.getApiVersion()).build(), + null, null, null); } catch (ApiException e) { - LOG.error("K8s submitter: delete persistent volume failed by {}, may cause some dirty data", - e.getMessage()); + throw new SubmarineRuntimeException(e.getCode(), e.getMessage()); } } - + private enum ParseOp { PARSE_OP_RESULT, PARSE_OP_DELETE diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java index 34b94df..d03138f 100644 --- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java +++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java @@ -28,6 +28,7 @@ import io.kubernetes.client.models.V1PodSpec; import io.kubernetes.client.models.V1ResourceRequirements; import io.kubernetes.client.models.V1Volume; import io.kubernetes.client.models.V1VolumeMount; +import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource; import org.apache.submarine.commons.utils.SubmarineConfVars; import org.apache.submarine.commons.utils.SubmarineConfiguration; @@ -39,6 +40,7 @@ import org.apache.submarine.server.api.spec.NotebookSpec; import org.apache.submarine.server.environment.EnvironmentManager; import org.apache.submarine.server.submitter.k8s.model.NotebookCR; import org.apache.submarine.server.submitter.k8s.model.NotebookCRSpec; +import org.apache.submarine.server.submitter.k8s.util.NotebookUtils; import java.util.ArrayList; import java.util.HashMap; @@ -149,7 +151,7 @@ public class NotebookSpecParser { List<V1VolumeMount> volumeMountList = new ArrayList<>(); V1VolumeMount volumeMount = new V1VolumeMount(); volumeMount.setMountPath(DEFAULT_MOUNT_PATH); - volumeMount.setName("notebook-pv-" + notebookSpec.getMeta().getName()); + volumeMount.setName(NotebookUtils.STORAGE_PREFIX + notebookSpec.getMeta().getName()); volumeMountList.add(volumeMount); container.setVolumeMounts(volumeMountList); @@ -159,8 +161,12 @@ public class NotebookSpecParser { // create volume object for persistent volume List<V1Volume> volumeList = new ArrayList<>(); V1Volume volume = new V1Volume(); - String volumeName = "notebook-pv-" + notebookSpec.getMeta().getName(); + String volumeName = NotebookUtils.STORAGE_PREFIX + notebookSpec.getMeta().getName(); + V1PersistentVolumeClaimVolumeSource persistentVolumeClaim = new V1PersistentVolumeClaimVolumeSource(); + String claimName = NotebookUtils.PVC_PREFIX + notebookSpec.getMeta().getName(); + persistentVolumeClaim.setClaimName(claimName); volume.setName(volumeName); + volume.setPersistentVolumeClaim(persistentVolumeClaim); volumeList.add(volume); podSpec.setVolumes(volumeList); diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java index 6e5803d..cde0cce 100644 --- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java +++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java @@ -57,7 +57,7 @@ public class VolumeSpecParser { } public static V1PersistentVolumeClaim parsePersistentVolumeClaim( - String name, String volume, String storage) { + String name, String scName, String storage) { V1PersistentVolumeClaim pvc = new V1PersistentVolumeClaim(); /* Required value @@ -73,10 +73,9 @@ public class VolumeSpecParser { pvc.setMetadata(pvcMetadata); V1PersistentVolumeClaimSpec pvcSpec = new V1PersistentVolumeClaimSpec(); - pvcSpec.setAccessModes(Collections.singletonList("ReadWriteMany")); - pvcSpec.setStorageClassName("standard"); + pvcSpec.setAccessModes(Collections.singletonList("ReadWriteOnce")); + pvcSpec.setStorageClassName(scName); pvcSpec.setResources(new V1ResourceRequirements().putRequestsItem("storage", new Quantity(storage))); - pvcSpec.setVolumeName(volume); // bind pvc to specific pv pvc.setSpec(pvcSpec); return pvc; diff --git a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java index ef4ed83..06996e7 100644 --- a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java +++ b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java @@ -44,6 +44,8 @@ public class NotebookUtils { private static final Logger LOG = LoggerFactory.getLogger(NotebookUtils.class); public static final String STORAGE = "10Gi"; + public static final String SC_NAME = "notebook-storageclass"; + public static final String STORAGE_PREFIX = "notebook-storage-"; public static final String PV_PREFIX = "notebook-pv-"; public static final String PVC_PREFIX = "notebook-pvc-"; public static final String HOST_PATH = "/mnt"; --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@submarine.apache.org For additional commands, e-mail: dev-h...@submarine.apache.org