This is an automated email from the ASF dual-hosted git repository.
pingsutw pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new 4b30e71 SUBMARINE-953. Submarine-server shouldn't create
persistentVolume manually for notebook
4b30e71 is described below
commit 4b30e71ec803e3c8209a73477fdea54d8a463869
Author: Kenchu123 <[email protected]>
AuthorDate: Sat Aug 7 18:53:51 2021 +0800
SUBMARINE-953. Submarine-server shouldn't create persistentVolume manually
for notebook
### What is this PR for?
<!-- A few sentences describing the overall goals of the pull request's
commits.
First time? Check out the contributing guide -
https://submarine.apache.org/contribution/contributions.html
-->
Source:
https://github.com/apache/submarine/blob/72be805fc4672f7f29c96cca95e16d59358d4dd2/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java#L379
We can find out that when we create a notebook custom resource, the server
will manually create the persistent volume for it.
However, this made submarine-server has to grant access to persistent
volume resources, which lead to setting cluster roles for the server.
(https://github.com/apache/submarine/blob/72be805fc4/helm-charts/submarine/templates/rbac.yaml#L61)
Since the server is in namespace scope, giving it cluster roles is
inappropriate. Besides, this makes multi-tenancy more difficult.
To fix the bug, we can create a storage class to dynamically provision
persistent volumes, and submarine-server will only need to deal with persistent
volume claim.
Ref:
- storage class:
https://kubernetes.io/docs/concepts/storage/storage-classes/#local
- notebook-cr spec: https://www.kubeflow.org/docs/reference/notebook/v1/
### What type of PR is it?
[Bug Fix]
### Todos
* [x] - add storageClass
* [x] - add persistentVolumeClaim spec to submarine-server
* [x] - remove submarine-server createPersistentVolume
* [x] - replace clutserrolebinding to rolebinding in helm-charts
* [x] - replace clusterrole to role in helm-charts
* [x] - modify submarine operator
### What is the Jira issue?
<!-- * Open an issue on Jira
https://issues.apache.org/jira/browse/SUBMARINE/
* Put link here, and add [SUBMARINE-*Jira number*] in PR title, eg.
`SUBMARINE-23. PR title`
-->
https://issues.apache.org/jira/browse/SUBMARINE-953
### How should this be tested?
<!--
* First time? Setup Travis CI as described on
https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed
behavior
* Outline any manual steps to test the PR here.
-->
Can be tested either using `helm install` or `submarine-operator`.
### Screenshots (if appropriate)
https://user-images.githubusercontent.com/17617373/128290829-ff38828c-3555-4c0c-b37d-6b147b762dde.mov
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No
Author: Kenchu123 <[email protected]>
Signed-off-by: Kevin <[email protected]>
Closes #697 from Kenchu123/SUBMARINE-953 and squashes the following commits:
c93d7124 [Kenchu123] SUBMARINE-953. Add Apache License to
helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml
877f8f2b [Kenchu123] SUBMARINE-953. Remove clusterrole and
clusterrolebinding in controller
6a1b8f1d [Kenchu123] SUBMARINE-953. Fix go format for controller.go
a5ad7430 [Kenchu123] SUBMARINE-953. remove quotation mark in helm-charts
notebook-controller storageclass
36713137 [Kenchu123] SUBMARINE-953. Change ClusterRole to Role in submarine
controller. Add storageClass to helm charts.
267c0714 [Kenchu123] SUBMARINE-953. Create Notebook with pvc and
storageClass, and remove pv creating and chage clusterrole to role
---
.../templates/storageclass.yaml | 24 ++++
helm-charts/submarine/templates/rbac.yaml | 7 +-
.../submarine/templates/submarine-server.yaml | 3 +-
submarine-cloud-v2/main.go | 4 +-
submarine-cloud-v2/pkg/controller/controller.go | 28 ++---
.../pkg/controller/submarine_server_rbac.go | 38 +++---
.../server/submitter/k8s/K8sSubmitter.java | 138 +++++++--------------
.../submitter/k8s/parser/NotebookSpecParser.java | 10 +-
.../submitter/k8s/parser/VolumeSpecParser.java | 7 +-
.../server/submitter/k8s/util/NotebookUtils.java | 2 +
10 files changed, 124 insertions(+), 137 deletions(-)
diff --git
a/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml
b/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml
new file mode 100644
index 0000000..de8a3d8
--- /dev/null
+++
b/helm-charts/submarine/charts/notebook-controller/templates/storageclass.yaml
@@ -0,0 +1,24 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+apiVersion: storage.k8s.io/v1
+kind: StorageClass
+metadata:
+ name: notebook-storageclass
+ labels:
+ app: notebook-controller
+provisioner: k8s.io/minikube-hostpath
+reclaimPolicy: Delete
diff --git a/helm-charts/submarine/templates/rbac.yaml
b/helm-charts/submarine/templates/rbac.yaml
index ff77fdf..fce2ef6 100644
--- a/helm-charts/submarine/templates/rbac.yaml
+++ b/helm-charts/submarine/templates/rbac.yaml
@@ -16,7 +16,7 @@
#
apiVersion: rbac.authorization.k8s.io/v1
-kind: ClusterRole
+kind: Role
metadata:
name: "{{ .Values.submarine.server.name }}"
rules:
@@ -58,7 +58,6 @@ rules:
- pods
- pods/log
- services
- - persistentvolumes
- persistentvolumeclaims
verbs:
- '*'
@@ -70,7 +69,7 @@ rules:
verbs:
- '*'
---
-kind: ClusterRoleBinding
+kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: "{{ .Values.submarine.server.name }}"
@@ -79,6 +78,6 @@ subjects:
namespace: {{ .Release.Namespace }}
name: "{{ .Values.submarine.server.name }}"
roleRef:
- kind: ClusterRole
+ kind: Role
name: "{{ .Values.submarine.server.name }}"
apiGroup: rbac.authorization.k8s.io
diff --git a/helm-charts/submarine/templates/submarine-server.yaml
b/helm-charts/submarine/templates/submarine-server.yaml
index 2452115..5a3f063 100644
--- a/helm-charts/submarine/templates/submarine-server.yaml
+++ b/helm-charts/submarine/templates/submarine-server.yaml
@@ -69,5 +69,4 @@ spec:
image: "{{ .Values.submarine.server.image }}"
imagePullPolicy: {{ .Values.submarine.server.imagePullPolicy }}
ports:
- - containerPort: 8080
-
+ - containerPort: 8080
\ No newline at end of file
diff --git a/submarine-cloud-v2/main.go b/submarine-cloud-v2/main.go
index 8bdf512..9db241c 100644
--- a/submarine-cloud-v2/main.go
+++ b/submarine-cloud-v2/main.go
@@ -95,8 +95,8 @@ func main() {
kubeInformerFactory.Core().V1().PersistentVolumeClaims(),
kubeInformerFactory.Extensions().V1beta1().Ingresses(),
traefikInformerFactory.Traefik().V1alpha1().IngressRoutes(),
- kubeInformerFactory.Rbac().V1().ClusterRoles(),
- kubeInformerFactory.Rbac().V1().ClusterRoleBindings(),
+ kubeInformerFactory.Rbac().V1().Roles(),
+ kubeInformerFactory.Rbac().V1().RoleBindings(),
submarineInformerFactory.Submarine().V1alpha1().Submarines())
// notice that there is no need to run Start methods in a separate
goroutine. (i.e. go kubeInformerFactory.Start(stopCh)
diff --git a/submarine-cloud-v2/pkg/controller/controller.go
b/submarine-cloud-v2/pkg/controller/controller.go
index 2c0ad80..d4af685 100644
--- a/submarine-cloud-v2/pkg/controller/controller.go
+++ b/submarine-cloud-v2/pkg/controller/controller.go
@@ -124,8 +124,8 @@ type Controller struct {
persistentvolumeclaimLister corelisters.PersistentVolumeClaimLister
ingressLister extlisters.IngressLister
ingressrouteLister traefiklisters.IngressRouteLister
- clusterroleLister rbaclisters.ClusterRoleLister
- clusterrolebindingLister rbaclisters.ClusterRoleBindingLister
+ roleLister rbaclisters.RoleLister
+ rolebindingLister rbaclisters.RoleBindingLister
// workqueue is a rate limited work queue. This is used to queue work
to be
// processed instead of performing it as soon as a change happens. This
// means we can ensure we only process a fixed amount of resources at a
@@ -153,8 +153,8 @@ func NewController(
persistentvolumeclaimInformer
coreinformers.PersistentVolumeClaimInformer,
ingressInformer extinformers.IngressInformer,
ingressrouteInformer traefikinformers.IngressRouteInformer,
- clusterroleInformer rbacinformers.ClusterRoleInformer,
- clusterrolebindingInformer rbacinformers.ClusterRoleBindingInformer,
+ roleInformer rbacinformers.RoleInformer,
+ rolebindingInformer rbacinformers.RoleBindingInformer,
submarineInformer informers.SubmarineInformer) *Controller {
// Add Submarine types to the default Kubernetes Scheme so Events can be
@@ -181,8 +181,8 @@ func NewController(
persistentvolumeclaimLister:
persistentvolumeclaimInformer.Lister(),
ingressLister: ingressInformer.Lister(),
ingressrouteLister: ingressrouteInformer.Lister(),
- clusterroleLister: clusterroleInformer.Lister(),
- clusterrolebindingLister:
clusterrolebindingInformer.Lister(),
+ roleLister: roleInformer.Lister(),
+ rolebindingLister: rolebindingInformer.Lister(),
workqueue:
workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(),
"Submarines"),
recorder: recorder,
incluster: incluster,
@@ -294,24 +294,24 @@ func NewController(
},
DeleteFunc: controller.handleObject,
})
-
clusterroleInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
+ roleInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
- newClusterRole := new.(*rbacv1.ClusterRole)
- oldClusterRole := old.(*rbacv1.ClusterRole)
- if newClusterRole.ResourceVersion ==
oldClusterRole.ResourceVersion {
+ newRole := new.(*rbacv1.Role)
+ oldRole := old.(*rbacv1.Role)
+ if newRole.ResourceVersion == oldRole.ResourceVersion {
return
}
controller.handleObject(new)
},
DeleteFunc: controller.handleObject,
})
-
clusterrolebindingInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
+
rolebindingInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: controller.handleObject,
UpdateFunc: func(old, new interface{}) {
- newClusterRoleBinding :=
new.(*rbacv1.ClusterRoleBinding)
- oldClusterRoleBinding :=
old.(*rbacv1.ClusterRoleBinding)
- if newClusterRoleBinding.ResourceVersion ==
oldClusterRoleBinding.ResourceVersion {
+ newRoleBinding := new.(*rbacv1.RoleBinding)
+ oldRoleBinding := old.(*rbacv1.RoleBinding)
+ if newRoleBinding.ResourceVersion ==
oldRoleBinding.ResourceVersion {
return
}
controller.handleObject(new)
diff --git a/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go
b/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go
index 431b30c..2b92b1e 100644
--- a/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go
+++ b/submarine-cloud-v2/pkg/controller/submarine_server_rbac.go
@@ -30,8 +30,8 @@ import (
"k8s.io/klog/v2"
)
-func newSubmarineServerClusterRole(submarine *v1alpha1.Submarine)
*rbacv1.ClusterRole {
- return &rbacv1.ClusterRole{
+func newSubmarineServerRole(submarine *v1alpha1.Submarine) *rbacv1.Role {
+ return &rbacv1.Role{
ObjectMeta: metav1.ObjectMeta{
Name: serverName,
OwnerReferences: []metav1.OwnerReference{
@@ -63,8 +63,8 @@ func newSubmarineServerClusterRole(submarine
*v1alpha1.Submarine) *rbacv1.Cluste
}
}
-func newSubmarineServerClusterRoleBinding(submarine *v1alpha1.Submarine)
*rbacv1.ClusterRoleBinding {
- return &rbacv1.ClusterRoleBinding{
+func newSubmarineServerRoleBinding(submarine *v1alpha1.Submarine)
*rbacv1.RoleBinding {
+ return &rbacv1.RoleBinding{
ObjectMeta: metav1.ObjectMeta{
Name: serverName,
OwnerReferences: []metav1.OwnerReference{
@@ -79,7 +79,7 @@ func newSubmarineServerClusterRoleBinding(submarine
*v1alpha1.Submarine) *rbacv1
},
},
RoleRef: rbacv1.RoleRef{
- Kind: "ClusterRole",
+ Kind: "Role",
Name: serverName,
APIGroup: "rbac.authorization.k8s.io",
},
@@ -91,12 +91,12 @@ func newSubmarineServerClusterRoleBinding(submarine
*v1alpha1.Submarine) *rbacv1
func (c *Controller) createSubmarineServerRBAC(submarine *v1alpha1.Submarine)
error {
klog.Info("[createSubmarineServerRBAC]")
- // Step1: Create ClusterRole
- clusterrole, err := c.clusterroleLister.Get(serverName)
+ // Step1: Create Role
+ role, err := c.roleLister.Roles(submarine.Namespace).Get(serverName)
// If the resource doesn't exist, we'll create it
if errors.IsNotFound(err) {
- clusterrole, err =
c.kubeclientset.RbacV1().ClusterRoles().Create(context.TODO(),
newSubmarineServerClusterRole(submarine), metav1.CreateOptions{})
- klog.Info(" Create ClusterRole: ", clusterrole.Name)
+ role, err =
c.kubeclientset.RbacV1().Roles(submarine.Namespace).Create(context.TODO(),
newSubmarineServerRole(submarine), metav1.CreateOptions{})
+ klog.Info(" Create Role: ", role.Name)
}
// If an error occurs during Get/Create, we'll requeue the item so we
can
@@ -106,28 +106,28 @@ func (c *Controller) createSubmarineServerRBAC(submarine
*v1alpha1.Submarine) er
return err
}
- if !metav1.IsControlledBy(clusterrole, submarine) {
- msg := fmt.Sprintf(MessageResourceExists, clusterrole.Name)
+ if !metav1.IsControlledBy(role, submarine) {
+ msg := fmt.Sprintf(MessageResourceExists, role.Name)
c.recorder.Event(submarine, corev1.EventTypeWarning,
ErrResourceExists, msg)
return fmt.Errorf(msg)
}
- clusterrolebinding, clusterrolebinding_err :=
c.clusterrolebindingLister.Get(serverName)
+ rolebinding, rolebinding_err :=
c.rolebindingLister.RoleBindings(submarine.Namespace).Get(serverName)
// If the resource doesn't exist, we'll create it
- if errors.IsNotFound(clusterrolebinding_err) {
- clusterrolebinding, clusterrolebinding_err =
c.kubeclientset.RbacV1().ClusterRoleBindings().Create(context.TODO(),
newSubmarineServerClusterRoleBinding(submarine), metav1.CreateOptions{})
- klog.Info(" Create ClusterRoleBinding: ",
clusterrolebinding.Name)
+ if errors.IsNotFound(rolebinding_err) {
+ rolebinding, rolebinding_err =
c.kubeclientset.RbacV1().RoleBindings(submarine.Namespace).Create(context.TODO(),
newSubmarineServerRoleBinding(submarine), metav1.CreateOptions{})
+ klog.Info(" Create RoleBinding: ", rolebinding.Name)
}
// If an error occurs during Get/Create, we'll requeue the item so we
can
// attempt processing again later. This could have been caused by a
// temporary network failure, or any other transient reason.
- if clusterrolebinding_err != nil {
- return clusterrolebinding_err
+ if rolebinding_err != nil {
+ return rolebinding_err
}
- if !metav1.IsControlledBy(clusterrolebinding, submarine) {
- msg := fmt.Sprintf(MessageResourceExists,
clusterrolebinding.Name)
+ if !metav1.IsControlledBy(rolebinding, submarine) {
+ msg := fmt.Sprintf(MessageResourceExists, rolebinding.Name)
c.recorder.Event(submarine, corev1.EventTypeWarning,
ErrResourceExists, msg)
return fmt.Errorf(msg)
}
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
index 23a862d..900419f 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/K8sSubmitter.java
@@ -39,9 +39,7 @@ import io.kubernetes.client.apis.CustomObjectsApi;
import io.kubernetes.client.models.V1DeleteOptionsBuilder;
import io.kubernetes.client.models.V1Deployment;
import io.kubernetes.client.models.V1ObjectMeta;
-import io.kubernetes.client.models.V1PersistentVolume;
import io.kubernetes.client.models.V1PersistentVolumeClaim;
-import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
import io.kubernetes.client.models.V1Pod;
import io.kubernetes.client.models.V1PodList;
import io.kubernetes.client.models.V1Service;
@@ -359,70 +357,67 @@ public class K8sSubmitter implements Submitter {
public Notebook createNotebook(NotebookSpec spec) throws
SubmarineRuntimeException {
Notebook notebook;
final String name = spec.getMeta().getName();
- final String pvName = NotebookUtils.PV_PREFIX + name;
+ final String scName = NotebookUtils.SC_NAME;
final String host = NotebookUtils.HOST_PATH;
final String storage = NotebookUtils.STORAGE;
final String pvcName = NotebookUtils.PVC_PREFIX + name;
String namespace = "default";
- NotebookCR notebookCR;
-
+
if (System.getenv(ENV_NAMESPACE) != null) {
namespace = System.getenv(ENV_NAMESPACE);
}
-
+
+ // parse notebook custom resource
+ NotebookCR notebookCR;
try {
- // create notebook custom resource
notebookCR = NotebookSpecParser.parseNotebook(spec);
Map<String, String> labels = new HashMap<>();
labels.put(NotebookCR.NOTEBOOK_OWNER_SELECTOR_KET,
spec.getMeta().getOwnerId());
notebookCR.getMetadata().setLabels(labels);
notebookCR.getMetadata().setNamespace(namespace);
-
- // create persistent volume
- createPersistentVolume(pvName, host, storage);
- } catch (ApiException e) {
- LOG.error("K8s submitter: Create persistent volume for Notebook object
failed by " + e.getMessage(), e);
- throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: Create
persistent volume for " +
- "Notebook object failed by " + e.getMessage());
} catch (JsonSyntaxException e) {
LOG.error("K8s submitter: parse response object failed by " +
e.getMessage(), e);
throw new SubmarineRuntimeException(500, "K8s Submitter parse upstream
response failed.");
}
-
+
// create persistent volume claim
try {
- createPersistentVolumeClaim(pvcName, namespace, pvName, storage);
+ createPersistentVolumeClaim(pvcName, namespace, scName, storage);
} catch (ApiException e) {
LOG.error("K8s submitter: Create persistent volume claim for Notebook
object failed by " +
e.getMessage(), e);
- rollbackCreationPV(pvName);
throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: Create
persistent volume claim for " +
"Notebook object failed by " + e.getMessage());
}
-
- // bind persistent volume claim
+
+ // create notebook custom resource
try {
- V1PersistentVolumeClaimVolumeSource pvcSource = new
V1PersistentVolumeClaimVolumeSource()
- .claimName(pvcName);
-
notebookCR.getSpec().getTemplate().getSpec().getVolumes().get(0).persistentVolumeClaim(pvcSource);
-
Object object = api.createNamespacedCustomObject(notebookCR.getGroup(),
notebookCR.getVersion(),
namespace, notebookCR.getPlural(), notebookCR, "true");
notebook = NotebookUtils.parseObject(object,
NotebookUtils.ParseOpt.PARSE_OPT_CREATE);
-
- // create Traefik custom resource
- createIngressRoute(notebookCR.getMetadata().getNamespace(),
notebookCR.getMetadata().getName());
-
} catch (JsonSyntaxException e) {
LOG.error("K8s submitter: parse response object failed by " +
e.getMessage(), e);
- rollbackCreationPVC(pvName, pvcName, namespace);
+ rollbackCreationPVC(pvcName, namespace);
throw new SubmarineRuntimeException(500, "K8s Submitter parse upstream
response failed.");
} catch (ApiException e) {
LOG.error("K8s submitter: parse Notebook object failed by " +
e.getMessage(), e);
- rollbackCreationPVC(pvName, pvcName, namespace);
+ rollbackCreationPVC(pvcName, namespace);
throw new SubmarineRuntimeException(e.getCode(), "K8s submitter: parse
Notebook object failed by " +
e.getMessage());
}
+
+ // create notebook Traefik custom resource
+ try {
+ createIngressRoute(notebookCR.getMetadata().getNamespace(),
notebookCR.getMetadata().getName());
+ } catch (ApiException e) {
+ LOG.error("K8s submitter: Create ingressroute for Notebook object failed
by " +
+ e.getMessage(), e);
+ rollbackCreationNotebook(notebookCR, namespace);
+ rollbackCreationPVC(pvcName, namespace);
+ throw new SubmarineRuntimeException(e.getCode(), "K8s submitter:
ingressroute for Notebook " +
+ "object failed by " + e.getMessage());
+ }
+
return notebook;
}
@@ -451,7 +446,6 @@ public class K8sSubmitter implements Submitter {
public Notebook deleteNotebook(NotebookSpec spec) throws
SubmarineRuntimeException {
Notebook notebook;
final String name = spec.getMeta().getName();
- final String pvName = NotebookUtils.PV_PREFIX + name;
final String pvcName = NotebookUtils.PVC_PREFIX + name;
String namespace = "default";
@@ -469,7 +463,6 @@ public class K8sSubmitter implements Submitter {
notebook = NotebookUtils.parseObject(object,
NotebookUtils.ParseOpt.PARSE_OPT_DELETE);
deleteIngressRoute(namespace, notebookCR.getMetadata().getName());
deletePersistentVolumeClaim(pvcName, namespace);
- deletePersistentVolume(pvName);
} catch (ApiException e) {
throw new SubmarineRuntimeException(e.getCode(), e.getMessage());
}
@@ -479,9 +472,16 @@ public class K8sSubmitter implements Submitter {
@Override
public List<Notebook> listNotebook(String id) throws
SubmarineRuntimeException {
List<Notebook> notebookList;
+
+ String namespace = "default";
+
+ if (System.getenv(ENV_NAMESPACE) != null) {
+ namespace = System.getenv(ENV_NAMESPACE);
+ }
+
try {
- Object object =
api.listClusterCustomObject(NotebookCR.CRD_NOTEBOOK_GROUP_V1,
- NotebookCR.CRD_NOTEBOOK_VERSION_V1,
NotebookCR.CRD_NOTEBOOK_PLURAL_V1,
+ Object object =
api.listNamespacedCustomObject(NotebookCR.CRD_NOTEBOOK_GROUP_V1,
+ NotebookCR.CRD_NOTEBOOK_VERSION_V1, namespace ,
NotebookCR.CRD_NOTEBOOK_PLURAL_V1,
"true", null, NotebookCR.NOTEBOOK_OWNER_SELECTOR_KET + "=" + id,
null, null, null);
notebookList = NotebookUtils.parseObjectForList(object);
@@ -557,49 +557,9 @@ public class K8sSubmitter implements Submitter {
}
}
- public void createPersistentVolume(String pvName, String hostPath, String
storage) throws ApiException {
- V1PersistentVolume pv = VolumeSpecParser.parsePersistentVolume(pvName,
hostPath, storage);
-
- try {
- V1PersistentVolume result = coreApi.createPersistentVolume(pv, "true",
null, null);
- } catch (ApiException e) {
- LOG.error("Exception when creating persistent volume " + e.getMessage(),
e);
- throw e;
- }
- }
-
- public void deletePersistentVolume(String pvName) throws ApiException {
- /*
- This version of Kubernetes-client/java has bug here.
- It will trigger exception as in
https://github.com/kubernetes-client/java/issues/86
- but it can still work fine and delete the PV.
- */
- try {
- V1Status result = coreApi.deletePersistentVolume(
- pvName, "true", null,
- null, null, null, null
- );
- } catch (ApiException e) {
- LOG.error("Exception when deleting persistent volume " + e.getMessage(),
e);
- throw e;
- } catch (JsonSyntaxException e) {
- if (e.getCause() instanceof IllegalStateException) {
- IllegalStateException ise = (IllegalStateException) e.getCause();
- if (ise.getMessage() != null && ise.getMessage().contains("Expected a
string but was BEGIN_OBJECT")) {
- LOG.debug("Catching exception because of issue " +
- "https://github.com/kubernetes-client/java/issues/86";, e);
- } else {
- throw e;
- }
- } else {
- throw e;
- }
- }
- }
-
- public void createPersistentVolumeClaim(String pvcName, String namespace,
String volume, String storage)
+ public void createPersistentVolumeClaim(String pvcName, String namespace,
String scName, String storage)
throws ApiException {
- V1PersistentVolumeClaim pvc =
VolumeSpecParser.parsePersistentVolumeClaim(pvcName, volume, storage);
+ V1PersistentVolumeClaim pvc =
VolumeSpecParser.parsePersistentVolumeClaim(pvcName, scName, storage);
try {
V1PersistentVolumeClaim result =
coreApi.createNamespacedPersistentVolumeClaim(
@@ -651,7 +611,7 @@ public class K8sSubmitter implements Submitter {
}
}
- private void createIngressRoute(String namespace, String name) {
+ private void createIngressRoute(String namespace, String name) throws
ApiException {
try {
IngressRoute ingressRoute = new IngressRoute();
V1ObjectMeta meta = new V1ObjectMeta();
@@ -706,30 +666,28 @@ public class K8sSubmitter implements Submitter {
return spec;
}
- private void rollbackCreationPV(String pvName) {
- try {
- deletePersistentVolume(pvName);
- } catch (ApiException e) {
- LOG.error("K8s submitter: delete persistent volume failed by {}, may
cause some dirty data",
- e.getMessage());
- }
- }
-
- private void rollbackCreationPVC(String pvName, String pvcName, String
namespace) {
+ private void rollbackCreationPVC(String pvcName, String namespace) {
try {
deletePersistentVolumeClaim(pvcName, namespace);
} catch (ApiException e) {
LOG.error("K8s submitter: delete persistent volume claim failed by {},
may cause some dirty data",
e.getMessage());
}
+ }
+
+ private void rollbackCreationNotebook(NotebookCR notebookCR, String
namespace)
+ throws SubmarineRuntimeException {
try {
- deletePersistentVolume(pvName);
+ Object object = api.deleteNamespacedCustomObject(notebookCR.getGroup(),
notebookCR.getVersion(),
+ namespace, notebookCR.getPlural(),
+ notebookCR.getMetadata().getName(),
+ new
V1DeleteOptionsBuilder().withApiVersion(notebookCR.getApiVersion()).build(),
+ null, null, null);
} catch (ApiException e) {
- LOG.error("K8s submitter: delete persistent volume failed by {}, may
cause some dirty data",
- e.getMessage());
+ throw new SubmarineRuntimeException(e.getCode(), e.getMessage());
}
}
-
+
private enum ParseOp {
PARSE_OP_RESULT,
PARSE_OP_DELETE
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java
index 34b94df..d03138f 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/NotebookSpecParser.java
@@ -28,6 +28,7 @@ import io.kubernetes.client.models.V1PodSpec;
import io.kubernetes.client.models.V1ResourceRequirements;
import io.kubernetes.client.models.V1Volume;
import io.kubernetes.client.models.V1VolumeMount;
+import io.kubernetes.client.models.V1PersistentVolumeClaimVolumeSource;
import org.apache.submarine.commons.utils.SubmarineConfVars;
import org.apache.submarine.commons.utils.SubmarineConfiguration;
@@ -39,6 +40,7 @@ import org.apache.submarine.server.api.spec.NotebookSpec;
import org.apache.submarine.server.environment.EnvironmentManager;
import org.apache.submarine.server.submitter.k8s.model.NotebookCR;
import org.apache.submarine.server.submitter.k8s.model.NotebookCRSpec;
+import org.apache.submarine.server.submitter.k8s.util.NotebookUtils;
import java.util.ArrayList;
import java.util.HashMap;
@@ -149,7 +151,7 @@ public class NotebookSpecParser {
List<V1VolumeMount> volumeMountList = new ArrayList<>();
V1VolumeMount volumeMount = new V1VolumeMount();
volumeMount.setMountPath(DEFAULT_MOUNT_PATH);
- volumeMount.setName("notebook-pv-" + notebookSpec.getMeta().getName());
+ volumeMount.setName(NotebookUtils.STORAGE_PREFIX +
notebookSpec.getMeta().getName());
volumeMountList.add(volumeMount);
container.setVolumeMounts(volumeMountList);
@@ -159,8 +161,12 @@ public class NotebookSpecParser {
// create volume object for persistent volume
List<V1Volume> volumeList = new ArrayList<>();
V1Volume volume = new V1Volume();
- String volumeName = "notebook-pv-" + notebookSpec.getMeta().getName();
+ String volumeName = NotebookUtils.STORAGE_PREFIX +
notebookSpec.getMeta().getName();
+ V1PersistentVolumeClaimVolumeSource persistentVolumeClaim = new
V1PersistentVolumeClaimVolumeSource();
+ String claimName = NotebookUtils.PVC_PREFIX +
notebookSpec.getMeta().getName();
+ persistentVolumeClaim.setClaimName(claimName);
volume.setName(volumeName);
+ volume.setPersistentVolumeClaim(persistentVolumeClaim);
volumeList.add(volume);
podSpec.setVolumes(volumeList);
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
index 6e5803d..cde0cce 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/parser/VolumeSpecParser.java
@@ -57,7 +57,7 @@ public class VolumeSpecParser {
}
public static V1PersistentVolumeClaim parsePersistentVolumeClaim(
- String name, String volume, String storage) {
+ String name, String scName, String storage) {
V1PersistentVolumeClaim pvc = new V1PersistentVolumeClaim();
/*
Required value
@@ -73,10 +73,9 @@ public class VolumeSpecParser {
pvc.setMetadata(pvcMetadata);
V1PersistentVolumeClaimSpec pvcSpec = new V1PersistentVolumeClaimSpec();
- pvcSpec.setAccessModes(Collections.singletonList("ReadWriteMany"));
- pvcSpec.setStorageClassName("standard");
+ pvcSpec.setAccessModes(Collections.singletonList("ReadWriteOnce"));
+ pvcSpec.setStorageClassName(scName);
pvcSpec.setResources(new
V1ResourceRequirements().putRequestsItem("storage", new Quantity(storage)));
- pvcSpec.setVolumeName(volume); // bind pvc to specific pv
pvc.setSpec(pvcSpec);
return pvc;
diff --git
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java
index ef4ed83..06996e7 100644
---
a/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java
+++
b/submarine-server/server-submitter/submitter-k8s/src/main/java/org/apache/submarine/server/submitter/k8s/util/NotebookUtils.java
@@ -44,6 +44,8 @@ public class NotebookUtils {
private static final Logger LOG =
LoggerFactory.getLogger(NotebookUtils.class);
public static final String STORAGE = "10Gi";
+ public static final String SC_NAME = "notebook-storageclass";
+ public static final String STORAGE_PREFIX = "notebook-storage-";
public static final String PV_PREFIX = "notebook-pv-";
public static final String PVC_PREFIX = "notebook-pvc-";
public static final String HOST_PATH = "/mnt";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
