This is an automated email from the ASF dual-hosted git repository.
pdallig pushed a commit to branch branch-0.10
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/branch-0.10 by this push:
new d96caee [ZEPPELIN-5445] Allow users to access the spark UI outside
the kubernetes cluster
d96caee is described below
commit d96caee43804230a163f5c73c92f604a4aacce02
Author: rick <[email protected]>
AuthorDate: Tue Jul 20 18:31:15 2021 +0800
[ZEPPELIN-5445] Allow users to access the spark UI outside the kubernetes
cluster
### What is this PR for?
Currently, when zeppelin server is running outside of k8s cluster, users
cannot access the spark UI directly from the web page of Zeppelin. So this PR
creates ingress resource in `k8s/interpreter/100-interpreter-spec.yaml` and
enables the `SPARK JOB` button to jump directly to the URL defined by ingress.
If user does not specify the property `zeppelin.spark.uiWebUrl`, it will be
automatically generated according to the template:
`{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}`, which is also the URL that the
`SPARK JOB` button points to. So this PR also configured the ingress.host by
this URL.
This PR also adds a property `zeppelin.k8s.spark.useIngress` to let user
choose whether to create ingress.
Example usage:
```
zeppelin.k8s.spark.useIngress true
```
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* <https://issues.apache.org/jira/browse/ZEPPELIN-5445>
### How should this be tested?
* CI pass and manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: rick <[email protected]>
Closes #4176 from rickchengx/ZEPPELIN-5445 and squashes the following
commits:
fdfe89ad3 [rick] update docs
1773d0657 [rick] [ZEPPELIN-5445] Allow users to access the spark UI outside
the kubernetes cluster
(cherry picked from commit ba766944b96308f117d488803a0d3a529b488dc6)
Signed-off-by: Philipp Dallig <[email protected]>
---
docs/quickstart/kubernetes.md | 4 +++
k8s/interpreter/100-interpreter-spec.yaml | 30 ++++++++++++++++++++++
.../launcher/K8sRemoteInterpreterProcess.java | 14 ++++++++++
3 files changed, 48 insertions(+)
diff --git a/docs/quickstart/kubernetes.md b/docs/quickstart/kubernetes.md
index 1c0b99a..7c7f88a 100644
--- a/docs/quickstart/kubernetes.md
+++ b/docs/quickstart/kubernetes.md
@@ -245,6 +245,7 @@ to customize,
4. Run a paragraph will create an interpreter using modified yaml files.
The interpreter pod can also be customized through the interpreter settings.
Here are some of the properties:
+
| Property Name | Default Value | Description |
| ----- | ----- | ----- |
| `zeppelin.k8s.namespace` | `default` | The Kubernetes namespace to use. |
@@ -256,6 +257,9 @@ The interpreter pod can also be customized through the
interpreter settings. Her
| `zeppelin.k8s.interpreter.imagePullSecrets` | (optional) | Set the
comma-separated list of Kubernetes secrets while pulling images, e.g.,
`mysecret1,mysecret2` |
| `zeppelin.k8s.interpreter.container.imagePullPolicy` | (optional) | Set the
pull policy of the interpreter image, e.g., `Always` |
| `zeppelin.k8s.spark.container.imagePullPolicy` | (optional) | Set the pull
policy of the spark image, e.g., `Always` |
+| `zeppelin.spark.uiWebUrl` | `//{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}`
| The URL for user to access Spark UI. The default value is a
[jinjava](https://github.com/HubSpot/jinjava) template that contains three
variables. |
+| `zeppelin.k8s.spark.useIngress` | (optional) | If true, the
[Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/)
will be created when creating the spark interpreter. So users can access the
Spark UI through Ingress. |
+| `zeppelin.k8s.spark.ingress.host` |
`{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}` | If
`zeppelin.k8s.spark.useIngress` is `true`, it configures the `host` value of
the Ingress. The default value is a
[jinjava](https://github.com/HubSpot/jinjava) template that contains three
variables. Users can access the Spark UI through a customized
`zeppelin.k8s.spark.ingress.host`. |
## Future work
diff --git a/k8s/interpreter/100-interpreter-spec.yaml
b/k8s/interpreter/100-interpreter-spec.yaml
index 1bf1615..91d6691 100644
--- a/k8s/interpreter/100-interpreter-spec.yaml
+++ b/k8s/interpreter/100-interpreter-spec.yaml
@@ -180,4 +180,34 @@ roleRef:
kind: Role
name: {{zeppelin.k8s.interpreter.pod.name}}
apiGroup: rbac.authorization.k8s.io
+{% if zeppelin.k8s.spark.useIngress is defined and
zeppelin.k8s.spark.useIngress == "true" %}
+---
+# create ingress of spark UI
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+ name: spark-ui-{{zeppelin.k8s.interpreter.pod.name}}
+ namespace: {{zeppelin.k8s.namespace}}
+ {% if zeppelin.k8s.server.uid is defined %}
+ ownerReferences:
+ - apiVersion: v1
+ controller: false
+ blockOwnerDeletion: false
+ kind: Pod
+ name: {{zeppelin.k8s.server.pod.name}}
+ uid: {{zeppelin.k8s.server.uid}}
+ {% endif %}
+spec:
+ rules:
+ - host: {{zeppelin.k8s.spark.ingress.host}}
+ http:
+ paths:
+ - pathType: Prefix
+ path: "/"
+ backend:
+ service:
+ name: {{zeppelin.k8s.interpreter.pod.name}}
+ port:
+ number: 4040
+{% endif %}
{% endif %}
\ No newline at end of file
diff --git
a/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
b/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
index bfd016f..c047ed7 100644
---
a/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
+++
b/zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
@@ -318,6 +318,20 @@ public class K8sRemoteInterpreterProcess extends
RemoteInterpreterManagedProcess
getPodName(),
k8sEnv.get(ENV_SERVICE_DOMAIN)
));
+
+ // configure interpreter property "zeppelin.k8s.spark.ingress.host" if
not defined, to enable spark ui through ingress
+ String ingressHost = (String)
properties.get("zeppelin.k8s.spark.ingress.host");
+ if (StringUtils.isBlank(ingressHost)) {
+ ingressHost = "{{PORT}}-{{SERVICE_NAME}}.{{SERVICE_DOMAIN}}";
+ }
+ properties.put("zeppelin.k8s.spark.ingress.host",
+ sparkUiWebUrlFromTemplate(
+ ingressHost,
+ webUiPort,
+ getPodName(),
+ k8sEnv.get(ENV_SERVICE_DOMAIN)
+ ));
+
// Resources of Interpreter Pod
if (properties.containsKey(SPARK_DRIVER_MEMORY)) {
String memory;
