This is an automated email from the ASF dual-hosted git repository.
liuxun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/submarine.git
The following commit(s) were added to refs/heads/master by this push:
new f98c6822 SUBMARINE-976. Configure ingress routing for services across
different namespaces (#985)
f98c6822 is described below
commit f98c6822137c05e09cf9378004fd2ac5d92567c8
Author: Josh Chuang <[email protected]>
AuthorDate: Tue Sep 13 18:53:19 2022 +0800
SUBMARINE-976. Configure ingress routing for services across different
namespaces (#985)
### What is this PR for?
<!-- A few sentences describing the overall goals of the pull request's
commits.
First time? Check out the contributing guide -
https://submarine.apache.org/contribution/contributions.html
-->
Configure Istio virtual service routing for submarine services across
different namespaces.
For example, a submarine workbench created by submitting `Submarine` custom
resource to k8s namespace `submarine-user-test`, can now only be connected to
with host `submarine-user-test.submarine` as the destination e.g. via URL
`http://submarine-user-test.submarine`.
To configure this, edit the newly added `spec.virtualservice.hosts` field
of the submarine CR and `spec.host` of the virtual service will be set
accordingly. For consistency, `spec.gateways` of the virtual service can now
also be set through editing the submarine CR.
### What type of PR is it?
Feature
### Todos
* [x] - update operator source file and artifact file
* [x] - add e2e test cases
* [x] - add new development doc
### What is the Jira issue?
<!-- * Open an issue on Jira
https://issues.apache.org/jira/browse/SUBMARINE/
* Put link here, and add [SUBMARINE-*Jira number*] in PR title, eg.
`SUBMARINE-23. PR title`
-->
https://issues.apache.org/jira/browse/SUBMARINE-976
### How should this be tested?
<!--
* First time? Setup Travis CI as described on
https://submarine.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed
behavior
* Outline any manual steps to test the PR here.
-->
Follow submarine-cloud-v3/docs/developer-guide.md
### Screenshots (if appropriate)
### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? Yes
* Does this need new documentation? Yes
---
submarine-cloud-v3/api/v1alpha1/submarine_types.go | 20 +-
.../api/v1alpha1/zz_generated.deepcopy.go | 30 +++
.../artifacts/submarine-virtualservice.yaml | 2 -
.../crd/bases/submarine.apache.org_submarines.yaml | 17 ++
.../config/samples/_v1alpha1_submarine.yaml | 5 +
.../controllers/submarine_controller_test.go | 202 ++++++++++++++++++---
.../controllers/submarine_virtualservice.go | 16 ++
submarine-cloud-v3/docs/developer-guide.md | 64 ++++++-
8 files changed, 313 insertions(+), 43 deletions(-)
diff --git a/submarine-cloud-v3/api/v1alpha1/submarine_types.go
b/submarine-cloud-v3/api/v1alpha1/submarine_types.go
index bac88cbd..5acc791f 100644
--- a/submarine-cloud-v3/api/v1alpha1/submarine_types.go
+++ b/submarine-cloud-v3/api/v1alpha1/submarine_types.go
@@ -46,6 +46,8 @@ type SubmarineSpec struct {
Server *SubmarineServerSpec `json:"server"`
// Database is the spec that defines the submarine database
Database *SubmarineDatabaseSpec `json:"database"`
+ // Virtualservice is the spec that defines the submarine virtualservice
+ Virtualservice *SubmarineVirtualserviceSpec `json:"virtualservice"`
// Tensorboard is the spec that defines the submarine tensorboard
Tensorboard *SubmarineTensorboardSpec `json:"tensorboard"`
// Mlflow is the spec that defines the submarine mlflow
@@ -54,7 +56,7 @@ type SubmarineSpec struct {
Minio *SubmarineMinioSpec `json:"minio"`
}
-// SubmarineServerSpec defins the desired submarine server
+// SubmarineServerSpec defines the desired submarine server
type SubmarineServerSpec struct {
// Image is the submarine server's docker image
Image string `json:"image"`
@@ -63,7 +65,7 @@ type SubmarineServerSpec struct {
Replicas *int32 `json:"replicas"`
}
-// SubmarineServerSpec defins the desired submarine database
+// SubmarineServerSpec defines the desired submarine database
type SubmarineDatabaseSpec struct {
// Image is the submarine database's docker image
Image string `json:"image"`
@@ -73,7 +75,15 @@ type SubmarineDatabaseSpec struct {
MysqlRootPasswordSecret string `json:"mysqlRootPasswordSecret"`
}
-// SubmarineServerSpec defins the desired submarine tensorboard
+// SubmarineVirtualserviceSpec defines the desired submarine virtualservice
+type SubmarineVirtualserviceSpec struct {
+ // Hosts is the submarine virtualservice's destination hosts
+ Hosts []string `json:"hosts,omitempty"`
+ // Hosts is the submarine virtualservice's gateways
+ Gateways []string `json:"gateways,omitempty"`
+}
+
+// SubmarineServerSpec defines the desired submarine tensorboard
type SubmarineTensorboardSpec struct {
// Enabled defines whether to enable tensorboard or not
Enabled *bool `json:"enabled"`
@@ -81,7 +91,7 @@ type SubmarineTensorboardSpec struct {
StorageSize string `json:"storageSize"`
}
-// SubmarineServerSpec defins the desired submarine mlflow
+// SubmarineServerSpec defines the desired submarine mlflow
type SubmarineMlflowSpec struct {
// Enabled defines whether to enable mlflow or not
Enabled *bool `json:"enabled"`
@@ -89,7 +99,7 @@ type SubmarineMlflowSpec struct {
StorageSize string `json:"storageSize"`
}
-// SubmarineServerSpec defins the desired submarine minio
+// SubmarineServerSpec defines the desired submarine minio
type SubmarineMinioSpec struct {
// Enabled defines whether to enable minio or not
Enabled *bool `json:"enabled"`
diff --git a/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
b/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
index 8bf981a5..bd02163c 100644
--- a/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
+++ b/submarine-cloud-v3/api/v1alpha1/zz_generated.deepcopy.go
@@ -172,6 +172,11 @@ func (in *SubmarineSpec) DeepCopyInto(out *SubmarineSpec) {
*out = new(SubmarineDatabaseSpec)
**out = **in
}
+ if in.Virtualservice != nil {
+ in, out := &in.Virtualservice, &out.Virtualservice
+ *out = new(SubmarineVirtualserviceSpec)
+ (*in).DeepCopyInto(*out)
+ }
if in.Tensorboard != nil {
in, out := &in.Tensorboard, &out.Tensorboard
*out = new(SubmarineTensorboardSpec)
@@ -249,3 +254,28 @@ func (in *SubmarineTensorboardSpec) DeepCopy()
*SubmarineTensorboardSpec {
in.DeepCopyInto(out)
return out
}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver,
writing into out. in must be non-nil.
+func (in *SubmarineVirtualserviceSpec) DeepCopyInto(out
*SubmarineVirtualserviceSpec) {
+ *out = *in
+ if in.Hosts != nil {
+ in, out := &in.Hosts, &out.Hosts
+ *out = make([]string, len(*in))
+ copy(*out, *in)
+ }
+ if in.Gateways != nil {
+ in, out := &in.Gateways, &out.Gateways
+ *out = make([]string, len(*in))
+ copy(*out, *in)
+ }
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver,
creating a new SubmarineVirtualserviceSpec.
+func (in *SubmarineVirtualserviceSpec) DeepCopy() *SubmarineVirtualserviceSpec
{
+ if in == nil {
+ return nil
+ }
+ out := new(SubmarineVirtualserviceSpec)
+ in.DeepCopyInto(out)
+ return out
+}
diff --git a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
index 6405ddea..d9c7ad81 100644
--- a/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
+++ b/submarine-cloud-v3/artifacts/submarine-virtualservice.yaml
@@ -21,8 +21,6 @@ kind: VirtualService
metadata:
name: submarine-virtual-service
spec:
- hosts:
- - "*"
gateways:
- submarine-cloud-v3-system/submarine-gateway
http:
diff --git
a/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
b/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
index 3db3c6b5..0cc6cf9f 100644
--- a/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
+++ b/submarine-cloud-v3/config/crd/bases/submarine.apache.org_submarines.yaml
@@ -128,6 +128,22 @@ spec:
version:
description: Version is the submarine docker image version
type: string
+ virtualservice:
+ description: Virtualservice is the spec that defines the
submarine
+ virtualservice
+ properties:
+ gateways:
+ description: Hosts is the submarine virtualservice's
gateways
+ items:
+ type: string
+ type: array
+ hosts:
+ description: Hosts is the submarine virtualservice's
destination
+ hosts
+ items:
+ type: string
+ type: array
+ type: object
required:
- database
- minio
@@ -135,6 +151,7 @@ spec:
- server
- tensorboard
- version
+ - virtualservice
type: object
status:
description: SubmarineStatus defines the observed state of
Submarine
diff --git a/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
b/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
index 8ae4c859..844f2afb 100644
--- a/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
+++ b/submarine-cloud-v3/config/samples/_v1alpha1_submarine.yaml
@@ -28,6 +28,11 @@ spec:
image: "apache/submarine:database-0.8.0-SNAPSHOT" # overwrite the image
when development
storageSize: "1Gi"
mysqlRootPasswordSecret: "root-pass-secret"
+ virtualservice:
+ hosts:
+ # - "yourHost" # configure when using custom hosts
+ gateways:
+ # - "yourNamespace/submarine-gateway" # configure when installing Helm
in a different namespace
tensorboard:
enabled: true
storageSize: "10Gi"
diff --git a/submarine-cloud-v3/controllers/submarine_controller_test.go
b/submarine-cloud-v3/controllers/submarine_controller_test.go
index 741e3e87..536bd98c 100644
--- a/submarine-cloud-v3/controllers/submarine_controller_test.go
+++ b/submarine-cloud-v3/controllers/submarine_controller_test.go
@@ -26,6 +26,7 @@ import (
. "github.com/onsi/ginkgo"
. "github.com/onsi/gomega"
"github.com/pkg/errors"
+ istiov1alpha3 "istio.io/client-go/pkg/apis/networking/v1alpha3"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -40,7 +41,11 @@ var _ = Describe("Submarine controller", func() {
// Define utility constants and variables
const (
- submarineNamespace = "submarine-test-submit-custom-resource" //
The namespace where the Submarine CR is created
+ // The namespaces where the Submarine CRs are created
+ submarineNamespaceDefaultCR = "submarine-test-submit-default-cr"
+ submarineNamespaceCustomCR = "submarine-test-submit-custom-cr"
+ customHost = "submarine-custom-host"
+ customGateway = "submarine-custom-gateway"
createNsTimeout = time.Second * 10
createNsInterval = time.Second * 2
@@ -54,19 +59,45 @@ var _ = Describe("Submarine controller", func() {
var (
// The name of Submarine is specified in the YAML file.
// Storing name to call k8sClient.Get with NamespacedName
- submarineName string
+ submarineNameDefaultCR string
+ submarineNameCustomCR string
+ submarineCustomHosts []string
+ submarineCustomGateways []string
ctx = context.Background()
)
- Context("Create a test namespace", func() {
- It("Should create a test namespace", func() {
- msg := fmt.Sprintf("Creating the test namespace %s",
submarineNamespace)
- By(msg)
+ Context("Create test namespaces", func() {
+ It(fmt.Sprintf("Should create namespace %s",
submarineNamespaceDefaultCR), func() {
+ By(fmt.Sprintf("Creating the test namespace %s",
submarineNamespaceDefaultCR))
+ ns := &corev1.Namespace{
+ ObjectMeta: metav1.ObjectMeta{
+ Name: submarineNamespaceDefaultCR, //
Namespace to test default CR
+ Labels: map[string]string{
+ "istio-injection": "enabled",
+ },
+ },
+ }
+ Expect(k8sClient.Create(ctx, ns)).Should(Succeed())
+ // We'll need to retry getting this newly created
namespace, given that creation may not immediately happen.
+ createdNs := &corev1.Namespace{} // stub
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespaceDefaultCR, Namespace: "default"},
createdNs)
+ if err != nil {
+ return false
+ }
+ return true
+ }, createNsTimeout, createNsInterval).Should(BeTrue())
+
+ // The namespace should have Istio label
+
Expect(createdNs.Labels["istio-injection"]).To(Equal("enabled"))
+ })
+ It(fmt.Sprintf("Should create namespace %s",
submarineNamespaceCustomCR), func() {
+ By(fmt.Sprintf("Creating the test namespace %s",
submarineNamespaceCustomCR))
ns := &corev1.Namespace{
ObjectMeta: metav1.ObjectMeta{
- Name: submarineNamespace,
+ Name: submarineNamespaceCustomCR, //
Namespace to test custom CR
Labels: map[string]string{
"istio-injection": "enabled",
},
@@ -77,7 +108,7 @@ var _ = Describe("Submarine controller", func() {
// We'll need to retry getting this newly created
namespace, given that creation may not immediately happen.
createdNs := &corev1.Namespace{} // stub
Eventually(func() bool {
- err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespace, Namespace: "default"}, createdNs)
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespaceCustomCR, Namespace: "default"},
createdNs)
if err != nil {
return false
}
@@ -89,24 +120,74 @@ var _ = Describe("Submarine controller", func() {
})
})
- Context("Create Submarine", func() {
- It("Should create a Submarine and it should become RUNNING",
func() {
- By("Creating a new Submarine")
+ Context("Create Submarines", func() {
+ It(fmt.Sprintf("Should create Submarine in %s and it should
become RUNNING", submarineNamespaceDefaultCR), func() {
+ By(fmt.Sprintf("Creating new Submarine in %s",
submarineNamespaceDefaultCR))
+ submarine, err :=
MakeSubmarineFromYaml("../config/samples/_v1alpha1_submarine.yaml")
+ Expect(err).To(BeNil())
+
+ // Leave Spec.Virtualservice.Host empty to test default
value
+ // Leave Spec.Virtualservice.Gateways empty to test
default value
+
+ // The name of Submarine is specified in the YAML file.
+ // Storing name to call k8sClient.Get with
NamespacedName
+ submarineNameDefaultCR = submarine.Name
+
+ // Create Submarines in our namespace
+ submarine.Namespace = submarineNamespaceDefaultCR
+ Expect(k8sClient.Create(ctx,
submarine)).Should(Succeed())
+
+ // We'll need to retry getting this newly created
Submarine, given that creation may not immediately happen.
+ createdSubmarine :=
&submarineapacheorgv1alpha1.Submarine{} // stub
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameDefaultCR, Namespace:
submarineNamespaceDefaultCR}, createdSubmarine)
+ if err != nil {
+ return false
+ }
+ return true
+ }, createNsTimeout, createNsInterval).Should(BeTrue())
+
+ // Wait for Submarine to be in RUNNING state
+ By(fmt.Sprintf("Waiting until Submarine %s/%s become
RUNNING", submarineNameDefaultCR, submarineNamespaceDefaultCR))
+ Eventually(func() bool {
+ err = k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameDefaultCR, Namespace:
submarineNamespaceDefaultCR}, createdSubmarine)
+ Expect(err).To(BeNil())
+
+ state :=
createdSubmarine.Status.SubmarineState.State
+
Expect(state).ToNot(Equal(submarineapacheorgv1alpha1.FailedState))
+ if createdSubmarine.Status.SubmarineState.State
== submarineapacheorgv1alpha1.RunningState {
+ return true
+ }
+ return false
+ }, createSubmarineTimeout,
createSubmarineInterval).Should(BeTrue())
+ })
+ It(fmt.Sprintf("Should create Submarine in %s and it should
become RUNNING", submarineNamespaceCustomCR), func() {
+ By(fmt.Sprintf("Creating new Submarine in %s",
submarineNamespaceCustomCR))
submarine, err :=
MakeSubmarineFromYaml("../config/samples/_v1alpha1_submarine.yaml")
Expect(err).To(BeNil())
+ // Set Spec.Virtualservice.Hosts to
[submarineCustomHosts] to test custom value
+ submarineCustomHosts = make([]string, 1, 1)
+ submarineCustomHosts[0] = customHost
+ submarine.Spec.Virtualservice.Hosts =
submarineCustomHosts
+
+ // Set Spec.Virtualservice.Gateways to
[submarineCustomGateway] to test custom value
+ submarineCustomGateways = make([]string, 1, 1)
+ submarineCustomGateways[0] = customGateway
+ submarine.Spec.Virtualservice.Gateways =
submarineCustomGateways
+
// The name of Submarine is specified in the YAML file.
// Storing name to call k8sClient.Get with
NamespacedName
- submarineName = submarine.Name
+ submarineNameCustomCR = submarine.Name
- // Create Submarine in our namespace
- submarine.Namespace = submarineNamespace
+ // Create Submarines in our namespace
+ submarine.Namespace = submarineNamespaceCustomCR
Expect(k8sClient.Create(ctx,
submarine)).Should(Succeed())
// We'll need to retry getting this newly created
Submarine, given that creation may not immediately happen.
createdSubmarine :=
&submarineapacheorgv1alpha1.Submarine{} // stub
Eventually(func() bool {
- err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineName, Namespace: submarineNamespace},
createdSubmarine)
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameCustomCR, Namespace:
submarineNamespaceCustomCR}, createdSubmarine)
if err != nil {
return false
}
@@ -114,10 +195,9 @@ var _ = Describe("Submarine controller", func() {
}, createNsTimeout, createNsInterval).Should(BeTrue())
// Wait for Submarine to be in RUNNING state
- msg := fmt.Sprintf("Waiting until Submarine %s/%s
become RUNNING", submarineName, submarineNamespace)
- By(msg)
+ By(fmt.Sprintf("Waiting until Submarine %s/%s become
RUNNING", submarineNameCustomCR, submarineNamespaceCustomCR))
Eventually(func() bool {
- err = k8sClient.Get(ctx,
types.NamespacedName{Name: submarineName, Namespace: submarineNamespace},
createdSubmarine)
+ err = k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameCustomCR, Namespace:
submarineNamespaceCustomCR}, createdSubmarine)
Expect(err).To(BeNil())
state :=
createdSubmarine.Status.SubmarineState.State
@@ -130,15 +210,62 @@ var _ = Describe("Submarine controller", func() {
})
})
+ Context("Verify Virtual Service Spec", func() {
+ It(fmt.Sprintf("Hosts and Gateways should have default values
In %s", submarineNamespaceDefaultCR), func() {
+ By(fmt.Sprintf("Getting Virtual Service In %s",
submarineNamespaceDefaultCR))
+ createdVirtualService :=
&istiov1alpha3.VirtualService{} // stub
+ err := k8sClient.Get(ctx, types.NamespacedName{Name:
virtualServiceName, Namespace: submarineNamespaceDefaultCR},
createdVirtualService)
+ Expect(err).To(BeNil())
+
+ // The default value for host is <submarine
namespace>.submarine
+
Expect(createdVirtualService.Spec.Hosts[0]).To(Equal(submarineNamespaceDefaultCR
+ ".submarine"))
+ // The default value for gateway is
submarine-cloud-v3-system/submarine-gateway
+
Expect(createdVirtualService.Spec.Gateways[0]).To(Equal("submarine-cloud-v3-system/submarine-gateway"))
+ })
+ It(fmt.Sprintf("Hosts and Gateways should have custom values In
%s", submarineNamespaceCustomCR), func() {
+ By(fmt.Sprintf("Getting Virtual Service In %s",
submarineNamespaceCustomCR))
+ createdVirtualService :=
&istiov1alpha3.VirtualService{} // stub
+ err := k8sClient.Get(ctx, types.NamespacedName{Name:
virtualServiceName, Namespace: submarineNamespaceCustomCR},
createdVirtualService)
+ Expect(err).To(BeNil())
+
+ // The custom value for hosts matches the submarine CR
+
Expect(createdVirtualService.Spec.Hosts).To(Equal(submarineCustomHosts))
+ // The custom value for gateways matches the submarine
CR
+
Expect(createdVirtualService.Spec.Gateways).To(Equal(submarineCustomGateways))
+ })
+ })
+
Context("Delete Submarine", func() {
- It("Should delete the Submarine", func() {
- Expect(submarineName).ToNot(BeNil())
+ It(fmt.Sprintf("Should delete the Submarine In %s",
submarineNamespaceDefaultCR), func() {
+ Expect(submarineNameDefaultCR).ToNot(BeNil())
+
+ By(fmt.Sprintf("Deleting Submarine %s/%s",
submarineNameDefaultCR, submarineNamespaceDefaultCR))
+ createdSubmarine :=
&submarineapacheorgv1alpha1.Submarine{} // stub
+ err := k8sClient.Get(ctx, types.NamespacedName{Name:
submarineNameDefaultCR, Namespace: submarineNamespaceDefaultCR},
createdSubmarine)
+ Expect(err).To(BeNil())
+
+ foreground := metav1.DeletePropagationForeground
+ err = k8sClient.Delete(ctx, createdSubmarine,
&client.DeleteOptions{
+ PropagationPolicy: &foreground,
+ })
+ Expect(err).To(BeNil())
- msg := fmt.Sprintf("Deleting Submarine %s/%s",
submarineName, submarineNamespace)
- By(msg)
+ // Wait for Submarine to be deleted entirely
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameDefaultCR, Namespace:
submarineNamespaceDefaultCR}, createdSubmarine)
+ if apierrors.IsNotFound(err) {
+ return true
+ }
+ Expect(err).To(BeNil())
+ return false
+ }, deleteSubmarineTimeout,
deleteSubmarineInterval).Should(BeTrue())
+ })
+ It(fmt.Sprintf("Should delete the Submarine In %s",
submarineNamespaceCustomCR), func() {
+ Expect(submarineNameCustomCR).ToNot(BeNil())
+ By(fmt.Sprintf("Deleting Submarine %s/%s",
submarineNameCustomCR, submarineNamespaceCustomCR))
createdSubmarine :=
&submarineapacheorgv1alpha1.Submarine{} // stub
- err := k8sClient.Get(ctx, types.NamespacedName{Name:
submarineName, Namespace: submarineNamespace}, createdSubmarine)
+ err := k8sClient.Get(ctx, types.NamespacedName{Name:
submarineNameCustomCR, Namespace: submarineNamespaceCustomCR}, createdSubmarine)
Expect(err).To(BeNil())
foreground := metav1.DeletePropagationForeground
@@ -149,7 +276,7 @@ var _ = Describe("Submarine controller", func() {
// Wait for Submarine to be deleted entirely
Eventually(func() bool {
- err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineName, Namespace: submarineNamespace},
createdSubmarine)
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNameCustomCR, Namespace:
submarineNamespaceCustomCR}, createdSubmarine)
if apierrors.IsNotFound(err) {
return true
}
@@ -160,18 +287,35 @@ var _ = Describe("Submarine controller", func() {
})
Context("Delete the test namespace", func() {
- It("Should delete the test namespace", func() {
- msg := fmt.Sprintf("Deleting the test namespace %s",
submarineNamespace)
- By(msg)
+ It(fmt.Sprintf("Should delete namespace %s",
submarineNamespaceDefaultCR), func() {
+ By(fmt.Sprintf("Deleting the test namespace %s",
submarineNamespaceDefaultCR))
+
+ createdNs := &corev1.Namespace{} // stub
+ Expect(k8sClient.Get(ctx, types.NamespacedName{Name:
submarineNamespaceDefaultCR, Namespace: "default"},
createdNs)).Should(Succeed())
+ Expect(k8sClient.Delete(ctx,
createdNs)).Should(Succeed())
+
+ // Wait for submarine to be deleted entirely
+
+ Eventually(func() bool {
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespaceDefaultCR, Namespace: "default"},
createdNs)
+ if apierrors.IsNotFound(err) {
+ return true
+ }
+ Expect(err).To(BeNil())
+ return false
+ }, deleteNsTimeout, deleteNsInterval).Should(BeTrue())
+ })
+ It(fmt.Sprintf("Should delete namespace %s",
submarineNamespaceCustomCR), func() {
+ By(fmt.Sprintf("Deleting the test namespace %s",
submarineNamespaceCustomCR))
createdNs := &corev1.Namespace{} // stub
- Expect(k8sClient.Get(ctx, types.NamespacedName{Name:
submarineNamespace, Namespace: "default"}, createdNs)).Should(Succeed())
+ Expect(k8sClient.Get(ctx, types.NamespacedName{Name:
submarineNamespaceCustomCR, Namespace: "default"}, createdNs)).Should(Succeed())
Expect(k8sClient.Delete(ctx,
createdNs)).Should(Succeed())
// Wait for submarine to be deleted entirely
Eventually(func() bool {
- err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespace, Namespace: "default"}, createdNs)
+ err := k8sClient.Get(ctx,
types.NamespacedName{Name: submarineNamespaceCustomCR, Namespace: "default"},
createdNs)
if apierrors.IsNotFound(err) {
return true
}
diff --git a/submarine-cloud-v3/controllers/submarine_virtualservice.go
b/submarine-cloud-v3/controllers/submarine_virtualservice.go
index 2158817e..b15a0d44 100644
--- a/submarine-cloud-v3/controllers/submarine_virtualservice.go
+++ b/submarine-cloud-v3/controllers/submarine_virtualservice.go
@@ -39,6 +39,22 @@ func (r *SubmarineReconciler) newSubmarineVirtualService(ctx
context.Context, su
r.Log.Error(err, "ParseVirtualService")
}
virtualService.Namespace = submarine.Namespace
+
+ virtualserviceHosts := submarine.Spec.Virtualservice.Hosts
+ if virtualserviceHosts != nil {
+ // Use `Hosts` defined in submarine spec
+ virtualService.Spec.Hosts = virtualserviceHosts
+ } else {
+ // Use default `<namespace>.submarine`
+ virtualService.Spec.Hosts = append(virtualService.Spec.Hosts,
submarine.Namespace+".submarine")
+ }
+
+ virtualserviceGateways := submarine.Spec.Virtualservice.Gateways
+ if virtualserviceGateways != nil {
+ // Use `Gateways` defined in submarine spec
+ virtualService.Spec.Gateways = virtualserviceGateways
+ }
+
err = controllerutil.SetControllerReference(submarine, virtualService,
r.Scheme)
if err != nil {
r.Log.Error(err, "Set VirtualService ControllerReference")
diff --git a/submarine-cloud-v3/docs/developer-guide.md
b/submarine-cloud-v3/docs/developer-guide.md
index 75badc02..9789824e 100644
--- a/submarine-cloud-v3/docs/developer-guide.md
+++ b/submarine-cloud-v3/docs/developer-guide.md
@@ -47,7 +47,7 @@ Before running submarine operator, install submarine
dependencies with helm. `--
helm install --set dev=true submarine ../helm-charts/submarine/ -n
submarine-cloud-v3-system
```
-Now we run the submarine operator.
+Run the submarine operator.
```bash
# Step1: Apply the submarine CRD.
@@ -62,17 +62,63 @@ make run
kubectl apply -n submarine-user-test -f config/samples/_v1alpha1_submarine.yaml
```
-If you follow the above steps, you can view the submarine workbench via the
same approach specified in the
[QuickStart](https://submarine.apache.org/docs/next/gettingStarted/quickstart)
section on the submarine website.
+Ensure that submarine is ready.
+```bash
+$ kubectl get pods -n submarine-cloud-v3-system
+NAME READY STATUS
RESTARTS AGE
+notebook-controller-deployment-5b489cf59d-52lff 1/1 Running 2
22h
+submarine-cloud-v3-controller-manager-7b5787d8bc-hvpbk 2/2 Running 5
22h
+training-operator-6dcd5b9c64-v7k7k 1/1 Running 1
22h
+
+$ kubectl get pods -n submarine-user-test
+NAME READY STATUS RESTARTS AGE
+submarine-database-0 2/2 Running 0 24m
+submarine-minio-6d757cc97c-qwft5 2/2 Running 0 24m
+submarine-mlflow-657f5f8f6-9zxt6 2/2 Running 0 24m
+submarine-server-6c787c69b5-pv2dr 2/2 Running 0 24m
+submarine-tensorboard-7b447d94dd-d6kkm 2/2 Running 0 24m
+
+$ kubectl get virtualservices -n submarine-user-test
+NAME GATEWAYS
HOSTS AGE
+submarine-virtual-service ["submarine-cloud-v3-system/submarine-gateway"]
["submarine-user-test.submarine"] 24m
+```
+
+The Istio virtual service now accepts traffic sent to destination `<submarine
namespace>.submarine`. In our case it's `submarine-user-test.submarine`. Expose
the service by forwarding traffic from local port 80 to the Istio ingress
gateway.
+
+```bash
+# You may need to grant kubectl the capacity to bind to previleged ports
without root.
+# sudo setcap CAP_NET_BIND_SERVICE=+ep <full path to kubectl binary>
+
+# Step4: Expose the service using kubectl port-forward
+kubectl port-forward --address 127.0.0.1 -n istio-system
service/istio-ingressgateway 80:80
+
+# Alternatively, use minikube tunnel, which asks for sudo and does not require
setcap.
+# It may provide an external IP address other than 127.0.0.1.
+# minikube tunnel
+# kubectl get service/istio-ingressgateway -n istio-system
+```
+
+For local development, resolve `submarine-user-test.submarine` to IP address
`127.0.0.1` by adding the following line to the file `~/etc/hosts`.
```bash
-# Step4: Cleanup submarine.
+127.0.0.1 submarine-user-test.submarine # For submarine local development
+```
+
+Now we can connect to the submarine workbench at
`http://submarine-user-test.submarine`.
+
+
+```bash
+# Step5: Stop exposing the service
+# Just close the running process at Step4.
+
+# Step6: Cleanup submarine.
kubectl delete -n submarine-user-test submarine example-submarine
-# Step5: Cleanup operator.
+# Step7: Cleanup operator.
# Just close the running process at Step2.
-# Step6: Delete the submarine CRD.
+# Step8: Delete the submarine CRD.
make uninstall
```
@@ -107,7 +153,7 @@ kubectl get deployment -n submarine-cloud-v3-system
# Step5: Deploy a submarine.
kubectl apply -n submarine-user-test -f config/samples/_v1alpha1_submarine.yaml
-# You can now view the submarine workbench
+# We can now expose the service like before and connect to the submarine
workbench
# Step6: Cleanup submarine.
kubectl delete -n submarine-user-test submarine example-submarine
@@ -122,7 +168,11 @@ make undeploy
### Installing Helm in a different namespace
-By default, the Istio virtual service created by the operator binds to the
Istio gateway `submarine-cloud-v3-system/submarine-gateway`, which should be
installed in the `submarine-cloud-v3-system` namespace via Helm. If `helm
install` is run with `-n <your_namespace>`, edit the spec `spec.gateways` of
the virtual service to be `<your_namespace>/submarine-gateway` manually once
it's created by the operator.
+By default, the Istio virtual service created by the operator binds to the
Istio gateway `submarine-cloud-v3-system/submarine-gateway`, which is installed
in the `submarine-cloud-v3-system` namespace via Helm. If `helm install` is run
with `-n <your_namespace>`, set `spec.virtualserice.gateways` of your submarine
CRs to `<your_namespace>/submarine-gateway`. Note that the data type is an
array of strings.
+
+### Use custom hosts
+
+You can set the desitination hosts of the Istio virtual service by setting
`spec.virtualservice.hosts` of the submarine CR. Note that the data type is an
array of strings. If that field is omitted, hosts defaults to
`["<namespace>.submarine"]` e.g. `["submarine-user-test.submarine"]`.
### Rebuild Operator Image
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
