[GitHub] [zeppelin] Reamer opened a new pull request #3825: [ZEPPELIN-4915] K8s with java lib
Reamer opened a new pull request #3825: URL: https://github.com/apache/zeppelin/pull/3825 ### What is this PR for? This PR changes from a Kubectl binary to a Java library for connecting to a Kubernetes cluster. I decided to use [fabric8/kubernetes-client](https://github.com/fabric8io/kubernetes-client), because it's also used by the [Apache Spark project](https://github.com/fabric8io/kubernetes-client#who-uses-kubernetes--openshift-java-client) and allows the import of our generic files rendered with jinja2. ### What type of PR is it? - Improvement ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-4915 ### How should this be tested? * Travis-CI: https://travis-ci.org/github/Reamer/zeppelin/builds/702285528 ### Questions: * Does the licenses files need update? Yes * Is there breaking changes for older versions? No * Does this needs documentation? No This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (ZEPPELIN-4916) Don't run note or paragraph when it is in running state
Jeff Zhang created ZEPPELIN-4916: Summary: Don't run note or paragraph when it is in running state Key: ZEPPELIN-4916 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4916 Project: Zeppelin Issue Type: Improvement Components: zeppelin-server Affects Versions: 0.9.0 Reporter: Jeff Zhang e.g. if one note is in running state via previous rest api call, then another attempt to run this note should not be allowed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3825: [ZEPPELIN-4915] K8s with java lib
zjffdu commented on a change in pull request #3825:
URL: https://github.com/apache/zeppelin/pull/3825#discussion_r446013880
##
File path: zeppelin-plugins/launcher/k8s-standard/pom.xml
##
@@ -37,21 +37,47 @@
Launcher/K8sStandardInterpreterLauncher
+4.10.2
+
+io.fabric8
+kubernetes-client
+${version.kubernetes.client}
Review comment:
how about `kubernetes.client.version` ? This is the convention we use
for other libraries.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3825: [ZEPPELIN-4915] K8s with java lib
zjffdu commented on a change in pull request #3825:
URL: https://github.com/apache/zeppelin/pull/3825#discussion_r446014487
##
File path: zeppelin-plugins/launcher/k8s-standard/pom.xml
##
@@ -37,21 +37,47 @@
Launcher/K8sStandardInterpreterLauncher
+4.10.2
+
+io.fabric8
+kubernetes-client
+${version.kubernetes.client}
+compile
+
+
com.hubspot.jinjava
jinjava
-2.4.12
+2.5.4
Review comment:
use property ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3825: [ZEPPELIN-4915] K8s with java lib
Reamer commented on a change in pull request #3825:
URL: https://github.com/apache/zeppelin/pull/3825#discussion_r446028539
##
File path: zeppelin-plugins/launcher/k8s-standard/pom.xml
##
@@ -37,21 +37,47 @@
Launcher/K8sStandardInterpreterLauncher
+4.10.2
+
+io.fabric8
+kubernetes-client
+${version.kubernetes.client}
Review comment:
accept the proposal
##
File path: zeppelin-plugins/launcher/k8s-standard/pom.xml
##
@@ -37,21 +37,47 @@
Launcher/K8sStandardInterpreterLauncher
+4.10.2
+
+io.fabric8
+kubernetes-client
+${version.kubernetes.client}
+compile
+
+
com.hubspot.jinjava
jinjava
-2.4.12
+2.5.4
Review comment:
accept the proposal
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [zeppelin] Reamer commented on a change in pull request #3825: [ZEPPELIN-4915] K8s with java lib
Reamer commented on a change in pull request #3825:
URL: https://github.com/apache/zeppelin/pull/3825#discussion_r446029378
##
File path:
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
##
@@ -102,76 +121,71 @@ public String getInterpreterSettingName() {
@Override
public void start(String userName) throws IOException {
-/**
- * If a spark interpreter process is running, userName is set in
preparation for --proxy-user
- */
-if (isUserImpersonatedForSpark &&
!StringUtils.containsIgnoreCase(userName, "anonymous") && isSpark()) {
- this.userName = userName;
-} else {
- this.userName = null;
-}
+
+Properties templateProperties = getTemplateBindings(userName);
// create new pod
-apply(specTempaltes, false);
-kubectl.wait(String.format("pod/%s", getPodName()), "condition=Ready",
getConnectTimeout()/1000);
+apply(specTempaltes, false, templateProperties);
if (portForward) {
podPort =
RemoteInterpreterUtils.findRandomAvailablePortOnAllLocalInterfaces();
- portForwardWatchdog = kubectl.portForward(
- String.format("pod/%s", getPodName()),
- new String[] {
- String.format("%s:%s", podPort, K8S_INTERPRETER_SERVICE_PORT)
- });
+ localPortForward =
client.pods().inNamespace(namespace).withName(podName).portForward(K8S_INTERPRETER_SERVICE_PORT,
podPort);
}
long startTime = System.currentTimeMillis();
+long timeoutTime = startTime + getConnectTimeout();
// wait until interpreter send started message through thrift rpc
synchronized (started) {
- if (!started.get()) {
+ while (!started.get()) {
+long timetoTimeout = timeoutTime - System.currentTimeMillis();
+if (timetoTimeout <= 0) {
+ stop();
+ throw new IOException("Launching zeppelin interpreter on kubernetes
is time out, kill it now");
Review comment:
@zjffdu How can the calling instance be informed that the creation of
the interpreting process has failed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3825: [ZEPPELIN-4915] K8s with java lib
zjffdu commented on a change in pull request #3825:
URL: https://github.com/apache/zeppelin/pull/3825#discussion_r446248037
##
File path:
zeppelin-plugins/launcher/k8s-standard/src/main/java/org/apache/zeppelin/interpreter/launcher/K8sRemoteInterpreterProcess.java
##
@@ -102,76 +121,71 @@ public String getInterpreterSettingName() {
@Override
public void start(String userName) throws IOException {
-/**
- * If a spark interpreter process is running, userName is set in
preparation for --proxy-user
- */
-if (isUserImpersonatedForSpark &&
!StringUtils.containsIgnoreCase(userName, "anonymous") && isSpark()) {
- this.userName = userName;
-} else {
- this.userName = null;
-}
+
+Properties templateProperties = getTemplateBindings(userName);
// create new pod
-apply(specTempaltes, false);
-kubectl.wait(String.format("pod/%s", getPodName()), "condition=Ready",
getConnectTimeout()/1000);
+apply(specTempaltes, false, templateProperties);
if (portForward) {
podPort =
RemoteInterpreterUtils.findRandomAvailablePortOnAllLocalInterfaces();
- portForwardWatchdog = kubectl.portForward(
- String.format("pod/%s", getPodName()),
- new String[] {
- String.format("%s:%s", podPort, K8S_INTERPRETER_SERVICE_PORT)
- });
+ localPortForward =
client.pods().inNamespace(namespace).withName(podName).portForward(K8S_INTERPRETER_SERVICE_PORT,
podPort);
}
long startTime = System.currentTimeMillis();
+long timeoutTime = startTime + getConnectTimeout();
// wait until interpreter send started message through thrift rpc
synchronized (started) {
- if (!started.get()) {
+ while (!started.get()) {
+long timetoTimeout = timeoutTime - System.currentTimeMillis();
+if (timetoTimeout <= 0) {
+ stop();
+ throw new IOException("Launching zeppelin interpreter on kubernetes
is time out, kill it now");
Review comment:
It depends on what kind of launcher it use. If it is the
StandardInterpreterLauncher, it would launch interpreter process via java
Process in `InterpreterProcessLauncher` where it would monitor the process
state.
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterManagedProcess.java#L116
Spark yarn cluster mode is a special case because the spark-submit process
will exit after submitting yarn app, so we would use `YarnAppMonitor` to
monitor the yarn app, and can detect the interpreter process fail earlier if
the yarn app is failed.
https://github.com/apache/zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/YarnAppMonitor.java#L83
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [zeppelin] alexanderswerdlow commented on pull request #2373: [ZEPPELIN-2598] Securing Zeppelin with OpenID Connect
alexanderswerdlow commented on pull request #2373: URL: https://github.com/apache/zeppelin/pull/2373#issuecomment-650245727 @alexott I have done so [here](https://issues.apache.org/jira/browse/ZEPPELIN-4901) but haven't received any response This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [Announce] New Zeppelin Committer: Philipp Dallig
It's exciting news because I was always impressed by high-quality code contributions from Philipp Dalling. Congratulations! Welcome aboard! On Thu, Jun 25, 2020 at 11:23 PM Jeff Zhang wrote: > > The Project Management Committee (PMC) for Apache Zeppelin > has invited Philipp Dallig to become a committer and we are very pleased > to announce that he has accepted. > > We greatly appreciate all of Philipp Dallig's hard work and generous > contributions to the project. We look forward to continued involvement in > the project. > > Congratulations & Welcome aboard Philipp Dallig ! > > > -- > Best Regards > > Jeff Zhang >
Re: [Announce] New Zeppelin Committer: Philipp Dallig
That's great news! Welcome aboard Philipp! On Fri, Jun 26, 2020 at 8:23 AM Jeff Zhang wrote: > > The Project Management Committee (PMC) for Apache Zeppelin > has invited Philipp Dallig to become a committer and we are very pleased > to announce that he has accepted. > > We greatly appreciate all of Philipp Dallig's hard work and generous > contributions to the project. We look forward to continued involvement in > the project. > > Congratulations & Welcome aboard Philipp Dallig ! > > > -- > Best Regards > > Jeff Zhang > -- With best wishes,Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)
[jira] [Created] (ZEPPELIN-4917) Interpreter process is not running
David Lacalle Castillo created ZEPPELIN-4917: Summary: Interpreter process is not running Key: ZEPPELIN-4917 URL: https://issues.apache.org/jira/browse/ZEPPELIN-4917 Project: Zeppelin Issue Type: Bug Components: Kubernetes Affects Versions: 0.9.0 Reporter: David Lacalle Castillo I have downloaded zeppeling-0.9.0-preview1 tar. I entered in k8s folder and in zeppelin-server.yaml I updated apache/zeppeling:0.9.0-SNAPSHOT by apache/zeppeling:0.9.0. After that I ran the next command: kubectl apply -f zeppelin-server.yaml I made a port forward and access zeppelin. Inside the notebook I am getting org.apache.zeppelin.interpreter.InterpreterException: java.io.IOException: Interpreter process is not running. This error comes with all examples. Looking at my kubernetes cluster the services for spark, python.. gets created. However, no pods gets created -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [Announce] New Zeppelin Committer: Philipp Dallig
welcome! On Saturday, June 27, 2020, Alex Ott wrote: > That's great news! Welcome aboard Philipp! > > On Fri, Jun 26, 2020 at 8:23 AM Jeff Zhang wrote: > > > > > The Project Management Committee (PMC) for Apache Zeppelin > > has invited Philipp Dallig to become a committer and we are very pleased > > to announce that he has accepted. > > > > We greatly appreciate all of Philipp Dallig's hard work and generous > > contributions to the project. We look forward to continued involvement in > > the project. > > > > Congratulations & Welcome aboard Philipp Dallig ! > > > > > > -- > > Best Regards > > > > Jeff Zhang > > > > > -- > With best wishes,Alex Ott > http://alexott.net/ > Twitter: alexott_en (English), alexott (Russian) >
