subject:"\[GitHub\] \[zeppelin\] liuxunorg commented on a change in pull request #3331\: \[Zeppelin\-4049\] Hadoop Submarine \(Machine Learning\) interpreter"

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-18 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r266725683
 
 

 ##
 File path: submarine/pom.xml
 ##
 @@ -0,0 +1,166 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+
+  
+zeppelin-interpreter-parent
+org.apache.zeppelin
+0.9.0-SNAPSHOT
+../zeppelin-interpreter-parent/pom.xml
+  
+
+  org.apache.zeppelin
+  zeppelin-submarine
+  jar
+  0.9.0-SNAPSHOT
+  Zeppelin: Submarine interpreter
+
+  
+
+submarine
+2.7.3
+2.4.0
+0.3.8
+20.0
+
+1.3
+  
+
+  
+
+  org.apache.zeppelin
+  zeppelin-python
+  0.9.0-SNAPSHOT
+
+
+  org.apache.zeppelin
+  zeppelin-shell
+  0.9.0-SNAPSHOT
+
+
+  org.hamcrest
+  hamcrest-all
+  ${hamcrest.all.version}
+  test
+
+
+  org.apache.hadoop
+  hadoop-common
+  ${hadoop.version}
+  
+
+  org.apache.commons
+  commons-compress
+
+
+  com.google.guava
+  guava
+
+
+  org.codehaus.jackson
+  jackson-mapper-asl
+
+
+  org.codehaus.jackson
+  jackson-xc
+
+
+  org.codehaus.jackson
+  jackson-jaxrs
+
+
+  org.codehaus.jackson
+  jackson-core-asl
+
+  
+
+
+  org.apache.hadoop
+  hadoop-hdfs
+  ${hadoop.version}
+  
+
+  com.google.guava
+  guava
+
+
+  io.netty
+  netty
+
+  
+
+
+  com.hubspot.jinjava
 
 Review comment:
   It's Apache License 2.0.
   https://github.com/HubSpot/jinjava/blob/master/LICENSE


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-18 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r266725506
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/IPySubmarineInterpreter.java
 ##
 @@ -0,0 +1,44 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.python.IPythonInterpreter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Properties;
+
+public class IPySubmarineInterpreter extends IPythonInterpreter {
+
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(IPySubmarineInterpreter.class);
+
+  private PySubmarineInterpreter pySubmarineInterpreter;
+
+  public IPySubmarineInterpreter(Properties properties) {
+super(properties);
+  }
+
+  @Override
+  public void open() throws InterpreterException {
+PySubmarineInterpreter pySparkInterpreter =
+
getInterpreterInTheSameSessionByClassName(PySubmarineInterpreter.class, false);
+
+pySubmarineInterpreter
 
 Review comment:
   Currently we are using the `pySubmarineInterpreter`
   Later, when we supported the `IpySubmarineInterpreter`, I will add it again.
   I refer to the code of the `Zeppelin Python Interpreter`,
   I am not familiar with `ipython` at present, the current code, It can be 
used normally,
   Therefore, in order to stabilize the system, still retain this part of the 
code.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-18 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r266724575
 
 

 ##
 File path: 
scripts/docker/submarine/1.0.0/zeppelin-cpu-submarine-interpreter/Dockerfile
 ##
 @@ -0,0 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM zeppelin-cpu-tensorflow_1.13.0-hadoop_3.1.2:1.0.0
 
 Review comment:
   `Zeppelin-cpu-tensorflow_1.13.0-hadoop_3.1.2` Is the basic image that we 
need to provide,
   Push to dockerhub for users to use, So now there is no dockerhub inside.
   The docker images for `zeppelin-cpu-tensorflow_1.13.0-hadoop_3.1.2` are in 
`scripts/docker/submarine/1.0.0/zeppelin-cpu-tensorflow_1.13.1-hadoop_3.1.2/Dockerfile`.
   Regarding the name of the image, felixcheung also made some comments.
   I want to do this:
    Base Docker Image:(Used to run the tensorflow algorithm in yarn)
   ```
   zeppelin hadoop submarine cpu base image name: zeppelin-submarine-cpu:1.0.0, 
Comment: tensorflow_1.13.0-cpu & hadoop_3.1.2
   zeppelin hadoop submarine gpu base image name: zeppelin-submarine-gpu:1.0.0, 
Comment: tensorflow_1.13.0-gpu & hadoop_3.1.2
   ```
   
    Interpreter Docker Image:(Used to run the submarine interpreter in yarn)
   ```
   zeppelin hadoop submarine Interpreter cpu image name: 
zeppelin-submarine-Interpreter-cpu:1.0.0, Comment: tensorflow_1.13.0-cpu & 
hadoop_3.1.2
   zeppelin hadoop submarine Interpreter gpu image name: 
zeppelin-submarine-Interpreter-gpu:1.0.0, Comment: tensorflow_1.13.0-gpu & 
hadoop_3.1.2
   ```
   
    Runtime Mount zeppelin lib
   There is no need to pre-install zeppelin in docker image.
   When zeppelin launches the submarine interpreter via `LaunchOnYarn`,
   Will mount the zeppelin submarine lib jar into the docker container.
   This very mode version of the update also reduces the size of the image


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-18 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r266718533
 
 

 ##
 File path: .travis.yml
 ##
 @@ -89,7 +89,7 @@ matrix:
 # Test interpreter modules
 - jdk: "oraclejdk8"
   dist: trusty
-  env: PYTHON="3" SPARKR="true" SCALA_VER="2.10" PROFILE="-Pscala-2.10" 
BUILD_FLAG="install -DskipTests -DskipRat" TEST_FLAG="test -DskipRat" 
MODULES="-pl $(echo 
.,zeppelin-interpreter,zeppelin-interpreter-api,${INTERPRETERS} | sed 
's/!//g')" TEST_PROJECTS=""
+  env: PYTHON="3" SPARKR="true" SCALA_VER="2.10" TENSORFLOW="1.0.0" 
PROFILE="-Pscala-2.10" BUILD_FLAG="install -DskipTests -DskipRat -am" 
TEST_FLAG="test -DskipRat" MODULES="-pl $(echo 
.,zeppelin-interpreter,zeppelin-interpreter-api,${INTERPRETERS} | sed 
's/!//g')" TEST_PROJECTS=""
 
 Review comment:
   In the PySubmarineInterpreterTest case 
`PySubmarineInterpreterTest::testTensorflow()`, just import the tensorflow 
library, then call tensorflow to get the version number function.
   If PySubmarineInterpreterTest fails, an exception is thrown. So the test 
case has nothing to do with the version of tensorflow.
   Official use is through 
`scripts/docker/submarine/1.0.0/zeppelin-cpu-tensorflow_1.13.1-hadoop_3.1.2/Dockerfile`.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-14 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265515902
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/commons/SubmarineConstants.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine.commons;
+
+/*
+ * NOTE: use lowercase + "_" for the option name
+ */
+public class SubmarineConstants {
+  // Docker container Environmental variable at `submarine-job-run-tf.jinja`
+  // and `/bin/interpreter.sh`
+  public static final String DOCKER_HADOOP_HDFS_HOME = 
"DOCKER_HADOOP_HDFS_HOME";
+  public static final String DOCKER_JAVA_HOME= "DOCKER_JAVA_HOME";
+  public static final String DOCKER_CONTAINER_TIME_ZONE = 
"DOCKER_CONTAINER_TIME_ZONE";
+  public static final String INTERPRETER_LAUNCH_MODE = 
"INTERPRETER_LAUNCH_MODE";
+
+  // interpreter.sh Environmental variable
+  public static final String SUBMARINE_HADOOP_HOME  = "SUBMARINE_HADOOP_HOME";
+  public static final String HADOOP_YARN_SUBMARINE_JAR  = 
"HADOOP_YARN_SUBMARINE_JAR";
+  public static final String SUBMARINE_INTERPRETER_DOCKER_IMAGE
+  = "SUBMARINE_INTERPRETER_DOCKER_IMAGE";
+
+  public static final String ZEPPELIN_SUBMARINE_AUTH_TYPE = 
"zeppelin.submarine.auth.type";
+  public static final String SUBMARINE_HADOOP_CONF_DIR  = 
"SUBMARINE_HADOOP_CONF_DIR";
+  public static final String SUBMARINE_HADOOP_KEYTAB= 
"SUBMARINE_HADOOP_KEYTAB";
+  public static final String SUBMARINE_HADOOP_PRINCIPAL = 
"SUBMARINE_HADOOP_PRINCIPAL";
+  public static final String SUBMARINE_HADOOP_KRB5_CONF = 
"submarine.hadoop.krb5.conf";
+
+  public static final String JOB_NAME = "JOB_NAME";
+  public static final String CLEAN_CHECKPOINT = "CLEAN_CHECKPOINT";
+  public static final String INPUT_PATH = "INPUT_PATH";
+  public static final String CHECKPOINT_PATH = "CHECKPOINT_PATH";
+  public static final String PS_LAUNCH_CMD = "PS_LAUNCH_CMD";
+  public static final String WORKER_LAUNCH_CMD = "WORKER_LAUNCH_CMD";
+  public static final String MACHINELEARING_DISTRIBUTED_ENABLE
+  = "machinelearing.distributed.enable";
+
+  public static final String ZEPPELIN_INTERPRETER_RPC_PORTRANGE
+  = "zeppelin.interpreter.rpc.portRange";
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-14 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265515861
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/commons/SubmarineConstants.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine.commons;
+
+/*
+ * NOTE: use lowercase + "_" for the option name
+ */
+public class SubmarineConstants {
+  // Docker container Environmental variable at `submarine-job-run-tf.jinja`
+  // and `/bin/interpreter.sh`
+  public static final String DOCKER_HADOOP_HDFS_HOME = 
"DOCKER_HADOOP_HDFS_HOME";
+  public static final String DOCKER_JAVA_HOME= "DOCKER_JAVA_HOME";
+  public static final String DOCKER_CONTAINER_TIME_ZONE = 
"DOCKER_CONTAINER_TIME_ZONE";
+  public static final String INTERPRETER_LAUNCH_MODE = 
"INTERPRETER_LAUNCH_MODE";
+
+  // interpreter.sh Environmental variable
+  public static final String SUBMARINE_HADOOP_HOME  = "SUBMARINE_HADOOP_HOME";
+  public static final String HADOOP_YARN_SUBMARINE_JAR  = 
"HADOOP_YARN_SUBMARINE_JAR";
+  public static final String SUBMARINE_INTERPRETER_DOCKER_IMAGE
+  = "SUBMARINE_INTERPRETER_DOCKER_IMAGE";
+
+  public static final String ZEPPELIN_SUBMARINE_AUTH_TYPE = 
"zeppelin.submarine.auth.type";
+  public static final String SUBMARINE_HADOOP_CONF_DIR  = 
"SUBMARINE_HADOOP_CONF_DIR";
+  public static final String SUBMARINE_HADOOP_KEYTAB= 
"SUBMARINE_HADOOP_KEYTAB";
+  public static final String SUBMARINE_HADOOP_PRINCIPAL = 
"SUBMARINE_HADOOP_PRINCIPAL";
+  public static final String SUBMARINE_HADOOP_KRB5_CONF = 
"submarine.hadoop.krb5.conf";
+
+  public static final String JOB_NAME = "JOB_NAME";
+  public static final String CLEAN_CHECKPOINT = "CLEAN_CHECKPOINT";
+  public static final String INPUT_PATH = "INPUT_PATH";
+  public static final String CHECKPOINT_PATH = "CHECKPOINT_PATH";
+  public static final String PS_LAUNCH_CMD = "PS_LAUNCH_CMD";
+  public static final String WORKER_LAUNCH_CMD = "WORKER_LAUNCH_CMD";
+  public static final String MACHINELEARING_DISTRIBUTED_ENABLE
+  = "machinelearing.distributed.enable";
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265411201
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/SubmarineInterpreter.java
 ##
 @@ -0,0 +1,275 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.display.ui.OptionInput.ParamOption;
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.apache.zeppelin.submarine.commons.SubmarineCommand;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.apache.zeppelin.submarine.commons.SubmarineUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.List;
+import java.util.Properties;
+
+import static 
org.apache.zeppelin.submarine.commons.SubmarineCommand.CLEAN_RUNTIME_CACHE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CLEAN_CHECKPOINT;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_CLEAN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_RUN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_SHOW;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_USAGE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.INPUT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.MACHINELEARING_DISTRIBUTED_ENABLE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.OPERATION_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.PS_LAUNCH_CMD;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_ALGORITHM_HDFS_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.TF_CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.USERNAME_SYMBOL;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.WORKER_LAUNCH_CMD;
+import static org.apache.zeppelin.submarine.commons.SubmarineUtils.unifyKey;
+
+/**
+ * SubmarineInterpreter of Hadoop Submarine implementation.
+ * Support for Hadoop Submarine cli. All the commands documented here
+ * 
https://github.com/apache/hadoop/tree/trunk/hadoop-submarine/hadoop-submarine-core
+ * /src/site/markdown/QuickStart.md is supported.
+ */
+public class SubmarineInterpreter extends Interpreter {
+  private Logger LOGGER = LoggerFactory.getLogger(SubmarineInterpreter.class);
+
+  // Number of submarines executed in parallel for each interpreter instance
+  protected int concurrentExecutedMax = 1;
+
+  private boolean needUpdateConfig = true;
+  private String currentReplName = "";
+
+  SubmarineContext submarineContext = null;
+
+  public SubmarineInterpreter(Properties properties) {
+super(properties);
+
+String concurrentMax = 
getProperty(SubmarineConstants.SUBMARINE_CONCURRENT_MAX, "1");
+concurrentExecutedMax = Integer.parseInt(concurrentMax);
+
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public void open() {
+LOGGER.info("SubmarineInterpreter open()");
+  }
+
+  @Override
+  public void close() {
+submarineContext.stopAllSubmarineJob();
+  }
+
+  private void setParagraphConfig(InterpreterContext context) {
+String replName = context.getReplName();
+if (StringUtils.equals(currentReplName, replName)) {
+  currentReplName = context.getReplName();
+  needUpdateConfig = true;
+}
+if (needUpdateConfig) {
+  needUpdateConfig = false;
+  if (currentReplName.equals("submarine") || currentReplName.isEmpty()) {
 
 Review comment:
   Ok, I am going to change the way to determine if it is a submarine

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265410629
 
 

 ##
 File path: scripts/docker/submarine/submarine-interpreter-cpu-1.0.0/Dockerfile
 ##
 @@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM zeppelin-tensorflow_1.10-hadoop_3.1.2-cpu:1.0.0
 
 Review comment:
   Yes, tensorflow uses version 1.13.0. I made a mistake and I will correct 
this error.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265410250
 
 

 ##
 File path: scripts/docker/submarine/submarine-interpreter-cpu-1.0.0/Dockerfile
 ##
 @@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM zeppelin-tensorflow_1.10-hadoop_3.1.2-cpu:1.0.0
 
 Review comment:
   About the Docker Image Name, Can you give me some suggestion? Thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265409886
 
 

 ##
 File path: 
scripts/docker/submarine/tensorflow_1.10-hadoop_3.1.2-gpu-1.0.0/Dockerfile
 ##
 @@ -0,0 +1,71 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04
+
+# Pick up some TF dependencies
+RUN apt-get update && apt-get install -y --allow-downgrades 
--no-install-recommends \
+  --allow-change-held-packages --allow-unauthenticated \
+  build-essential libfreetype6-dev libpng12-dev \
+  libzmq3-dev pkg-config python python-dev \
+  rsync software-properties-common curl unzip wget grep sed vim \
+iputils-ping net-tools gdb python2.7-dbg tzdata \
+cuda-cublas-9-0 cuda-cufft-9-0 cuda-curand-9-0 cuda-cusolver-9-0 \
+  cuda-cusparse-9-0 libcudnn7=7.0.5.15-1+cuda9.0 && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install 
-yq krb5-user libpam-krb5 && apt-get clean
+
+RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
+  python get-pip.py && \
+  rm get-pip.py
+
+RUN echo "Install python related packages" && \
+  apt-get -y update && \
+  apt-get install -y gfortran && \
+  # numerical/algebra packages
+  apt-get install -y libblas-dev libatlas-dev liblapack-dev && \
+  # font, image for matplotlib
+  apt-get install -y libpng-dev libxft-dev && \
+  # for tkinter
+  apt-get install -y python-tk libxml2-dev libxslt-dev zlib1g-dev
+
+RUN pip --no-cache-dir install Pillow h5py ipykernel jupyter matplotlib numpy 
pandas scipy sklearn && \
+  python -m ipykernel.kernelspec
+
+# Set the locale
+# disable bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
+RUN apt-get clean && apt-get update && apt-get install -y locales
+RUN locale-gen en_US.UTF-8
+
+# Install TensorFlow GPU version.
+ENV TENSORFLOW_VERSION="1.10.0"
+RUN pip --no-cache-dir install \
+  
http://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-${TENSORFLOW_VERSION}-cp27-none-linux_x86_64.whl
+RUN apt-get update && apt-get install git -y
+
+# Install hadoop
+ENV HADOOP_VERSION="3.1.2"
+RUN wget 
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
 
 Review comment:
   OK, Thanks for your correction. :-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265402272
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/commons/SubmarineConstants.java
 ##
 @@ -0,0 +1,119 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine.commons;
+
+/*
+ * NOTE: use lowercase + "_" for the option name
+ */
+public class SubmarineConstants {
+  // Docker container Environmental variable at `submarine-job-run-tf.jinja`
+  // and `/bin/interpreter.sh`
+  public static final String DOCKER_HADOOP_HDFS_HOME = 
"DOCKER_HADOOP_HDFS_HOME";
+  public static final String DOCKER_JAVA_HOME= "DOCKER_JAVA_HOME";
+  public static final String DOCKER_CONTAINER_TIME_ZONE = 
"DOCKER_CONTAINER_TIME_ZONE";
+  public static final String INTERPRETER_LAUNCH_MODE = 
"INTERPRETER_LAUNCH_MODE";
+
+  // interpreter.sh Environmental variable
+  public static final String SUBMARINE_HADOOP_HOME  = "SUBMARINE_HADOOP_HOME";
+  public static final String HADOOP_YARN_SUBMARINE_JAR  = 
"HADOOP_YARN_SUBMARINE_JAR";
+  public static final String SUBMARINE_INTERPRETER_DOCKER_IMAGE
+  = "SUBMARINE_INTERPRETER_DOCKER_IMAGE";
+
+  public static final String ZEPPELIN_SUBMARINE_AUTH_TYPE = 
"zeppelin.submarine.auth.type";
+  public static final String SUBMARINE_HADOOP_CONF_DIR  = 
"SUBMARINE_HADOOP_CONF_DIR";
+  public static final String SUBMARINE_HADOOP_KEYTAB= 
"SUBMARINE_HADOOP_KEYTAB";
+  public static final String SUBMARINE_HADOOP_PRINCIPAL = 
"SUBMARINE_HADOOP_PRINCIPAL";
+  public static final String SUBMARINE_HADOOP_KRB5_CONF = 
"submarine.hadoop.krb5.conf";
+
+  public static final String JOB_NAME = "JOB_NAME";
+  public static final String CLEAN_CHECKPOINT = "CLEAN_CHECKPOINT";
+  public static final String INPUT_PATH = "INPUT_PATH";
+  public static final String CHECKPOINT_PATH = "CHECKPOINT_PATH";
+  public static final String PS_LAUNCH_CMD = "PS_LAUNCH_CMD";
+  public static final String WORKER_LAUNCH_CMD = "WORKER_LAUNCH_CMD";
+  public static final String MACHINELEARING_DISTRIBUTED_ENABLE
+  = "machinelearing.distributed.enable";
+
+  public static final String ZEPPELIN_INTERPRETER_RPC_PORTRANGE
+  = "zeppelin.interpreter.rpc.portRange";
 
 Review comment:
   This definition is used in the second PR (`YarnInterpreterLauncher`), This 
PR is merged and will be submitted shortly.
   1. The `submarine interpreter` runs in a remote Docker container via `YARN`.
   2. Work together through this PRC port with the local zeppelin server.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265401783
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/SubmarineShellInterpreter.java
 ##
 @@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.commons.exec.CommandLine;
+import org.apache.commons.exec.DefaultExecutor;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.shell.ShellInterpreter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Properties;
+
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_ALGORITHM_HDFS_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_HADOOP_KEYTAB;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_HADOOP_PRINCIPAL;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.TF_CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.USERNAME_SYMBOL;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.ZEPPELIN_SUBMARINE_AUTH_TYPE;
+
+/**
+ * Submarine Shell interpreter for Zeppelin.
+ */
+public class SubmarineShellInterpreter extends ShellInterpreter {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(SubmarineShellInterpreter.class);
+
+  private final boolean isWindows = 
System.getProperty("os.name").startsWith("Windows");
+  private final String shell = isWindows ? "cmd /c" : "bash -c";
+
+  public SubmarineShellInterpreter(Properties property) {
+super(property);
+  }
+
+  @Override
+  public InterpreterResult internalInterpret(String cmd, InterpreterContext 
intpContext) {
+// algorithm path & checkpoint path support replaces ${username} with real 
user name
+String algorithmPath = 
properties.getProperty(SUBMARINE_ALGORITHM_HDFS_PATH, "");
+if (algorithmPath.contains(USERNAME_SYMBOL)) {
+  algorithmPath = algorithmPath.replace(USERNAME_SYMBOL, userName);
+  properties.setProperty(SUBMARINE_ALGORITHM_HDFS_PATH, algorithmPath);
+}
+String checkpointPath = properties.getProperty(
+TF_CHECKPOINT_PATH, "");
+if (checkpointPath.contains(USERNAME_SYMBOL)) {
+  checkpointPath = checkpointPath.replace(USERNAME_SYMBOL, userName);
+  properties.setProperty(TF_CHECKPOINT_PATH, checkpointPath);
+}
+
+return super.internalInterpret(cmd, intpContext);
+  }
+
+  @Override
+  protected boolean runKerberosLogin() {
+try {
+  createSecureConfiguration();
+  return true;
+} catch (Exception e) {
+  LOGGER.error("Unable to run kinit for zeppelin", e);
+}
+return false;
+  }
+
+  public void createSecureConfiguration() throws InterpreterException {
+Properties properties = getProperties();
+CommandLine cmdLine = CommandLine.parse(shell);
+cmdLine.addArgument("-c", false);
+String kinitCommand = String.format("kinit -k -t %s %s",
+properties.getProperty(SUBMARINE_HADOOP_KEYTAB),
+properties.getProperty(SUBMARINE_HADOOP_PRINCIPAL));
+cmdLine.addArgument(kinitCommand, false);
+DefaultExecutor executor = new DefaultExecutor();
+try {
+  executor.execute(cmdLine);
+} catch (Exception e) {
+  LOGGER.error("Unable to run kinit for zeppelin user " + kinitCommand, e);
+  throw new InterpreterException(e);
+}
+  }
+
+  @Override
+  protected boolean isKerboseEnabled() {
+if (!StringUtils.isAnyEmpty(getProperty(ZEPPELIN_SUBMARINE_AUTH_TYPE))
+&& getProperty(ZEPPELIN_SUBMARINE_AUTH_TYPE)
+.equalsIgnoreCase("kerberos")) {
 
 Review comment:
   OK.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265401655
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/SubmarineInterpreter.java
 ##
 @@ -0,0 +1,275 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.display.ui.OptionInput.ParamOption;
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.apache.zeppelin.submarine.commons.SubmarineCommand;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.apache.zeppelin.submarine.commons.SubmarineUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.List;
+import java.util.Properties;
+
+import static 
org.apache.zeppelin.submarine.commons.SubmarineCommand.CLEAN_RUNTIME_CACHE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CLEAN_CHECKPOINT;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_CLEAN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_RUN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_SHOW;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_USAGE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.INPUT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.MACHINELEARING_DISTRIBUTED_ENABLE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.OPERATION_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.PS_LAUNCH_CMD;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_ALGORITHM_HDFS_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.TF_CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.USERNAME_SYMBOL;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.WORKER_LAUNCH_CMD;
+import static org.apache.zeppelin.submarine.commons.SubmarineUtils.unifyKey;
+
+/**
+ * SubmarineInterpreter of Hadoop Submarine implementation.
+ * Support for Hadoop Submarine cli. All the commands documented here
+ * 
https://github.com/apache/hadoop/tree/trunk/hadoop-submarine/hadoop-submarine-core
+ * /src/site/markdown/QuickStart.md is supported.
+ */
+public class SubmarineInterpreter extends Interpreter {
+  private Logger LOGGER = LoggerFactory.getLogger(SubmarineInterpreter.class);
+
+  // Number of submarines executed in parallel for each interpreter instance
+  protected int concurrentExecutedMax = 1;
+
+  private boolean needUpdateConfig = true;
+  private String currentReplName = "";
+
+  SubmarineContext submarineContext = null;
+
+  public SubmarineInterpreter(Properties properties) {
+super(properties);
+
+String concurrentMax = 
getProperty(SubmarineConstants.SUBMARINE_CONCURRENT_MAX, "1");
+concurrentExecutedMax = Integer.parseInt(concurrentMax);
+
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public void open() {
+LOGGER.info("SubmarineInterpreter open()");
+  }
+
+  @Override
+  public void close() {
+submarineContext.stopAllSubmarineJob();
+  }
+
+  private void setParagraphConfig(InterpreterContext context) {
+String replName = context.getReplName();
+if (StringUtils.equals(currentReplName, replName)) {
+  currentReplName = context.getReplName();
+  needUpdateConfig = true;
+}
+if (needUpdateConfig) {
+  needUpdateConfig = false;
+  if (currentReplName.equals("submarine") || currentReplName.isEmpty()) {
+context.getConfig().put("editorHide", true);
+context.getConfig().put("title",

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265398900
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/SubmarineInterpreter.java
 ##
 @@ -0,0 +1,275 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.display.ui.OptionInput.ParamOption;
+import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
+import org.apache.zeppelin.scheduler.Scheduler;
+import org.apache.zeppelin.scheduler.SchedulerFactory;
+import org.apache.zeppelin.submarine.commons.SubmarineCommand;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.apache.zeppelin.submarine.commons.SubmarineUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.List;
+import java.util.Properties;
+
+import static 
org.apache.zeppelin.submarine.commons.SubmarineCommand.CLEAN_RUNTIME_CACHE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.CLEAN_CHECKPOINT;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_CLEAN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_RUN;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_JOB_SHOW;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.COMMAND_USAGE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.INPUT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.MACHINELEARING_DISTRIBUTED_ENABLE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.OPERATION_TYPE;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.PS_LAUNCH_CMD;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.SUBMARINE_ALGORITHM_HDFS_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.TF_CHECKPOINT_PATH;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.USERNAME_SYMBOL;
+import static 
org.apache.zeppelin.submarine.commons.SubmarineConstants.WORKER_LAUNCH_CMD;
+import static org.apache.zeppelin.submarine.commons.SubmarineUtils.unifyKey;
+
+/**
+ * SubmarineInterpreter of Hadoop Submarine implementation.
+ * Support for Hadoop Submarine cli. All the commands documented here
+ * 
https://github.com/apache/hadoop/tree/trunk/hadoop-submarine/hadoop-submarine-core
+ * /src/site/markdown/QuickStart.md is supported.
+ */
+public class SubmarineInterpreter extends Interpreter {
+  private Logger LOGGER = LoggerFactory.getLogger(SubmarineInterpreter.class);
+
+  // Number of submarines executed in parallel for each interpreter instance
+  protected int concurrentExecutedMax = 1;
+
+  private boolean needUpdateConfig = true;
+  private String currentReplName = "";
+
+  SubmarineContext submarineContext = null;
+
+  public SubmarineInterpreter(Properties properties) {
+super(properties);
+
+String concurrentMax = 
getProperty(SubmarineConstants.SUBMARINE_CONCURRENT_MAX, "1");
+concurrentExecutedMax = Integer.parseInt(concurrentMax);
+
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public void open() {
+LOGGER.info("SubmarineInterpreter open()");
+  }
+
+  @Override
+  public void close() {
+submarineContext.stopAllSubmarineJob();
+  }
+
+  private void setParagraphConfig(InterpreterContext context) {
+String replName = context.getReplName();
+if (StringUtils.equals(currentReplName, replName)) {
+  currentReplName = context.getReplName();
+  needUpdateConfig = true;
+}
+if (needUpdateConfig) {
+  needUpdateConfig = false;
+  if (currentReplName.equals("submarine") || currentReplName.isEmpty()) {
 
 Review comment:
   I am at https://github.com/apache/zeppelin/pull/3271 , Has all interpreters

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265397993
 
 

 ##
 File path: scripts/docker/submarine/submarine-interpreter-cpu-1.0.0/Dockerfile
 ##
 @@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM zeppelin-tensorflow_1.10-hadoop_3.1.2-cpu:1.0.0
 
 Review comment:
   1. I have carefully considered the issue of the docker name. Otherwise it 
will be difficult to maintain in the future.
   2. Because the name needs to include the name and version number of the main 
component in the image.
   3. **Naming rules**
   
   + The name `_` indicates the separator, splitting different systems, for 
example: `tensorflow` and `hadoop`
   + The name`-` indicates the connector, the connection system and the 
version number, for example: `hadoop-3.1.2`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-13 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r265397174
 
 

 ##
 File path: 
scripts/docker/submarine/tensorflow_1.10-hadoop_3.12-cpu-1.0.0/Dockerfile
 ##
 @@ -0,0 +1,69 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM ubuntu:16.04
+
+# Pick up some TF dependencies
+RUN apt-get update && apt-get install -y --allow-downgrades 
--no-install-recommends \
+--allow-change-held-packages --allow-unauthenticated \
+build-essential libfreetype6-dev libpng12-dev \
+libzmq3-dev pkg-config python python-dev \
+rsync software-properties-common curl unzip wget grep sed vim iputils-ping 
net-tools gdb python2.7-dbg tzdata && \
+  apt-get clean && \
+  rm -rf /var/lib/apt/lists/*
+
+RUN export DEBIAN_FRONTEND=noninteractive && apt-get update && apt-get install 
-yq krb5-user libpam-krb5 && apt-get clean
+
+RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
+  python get-pip.py && \
+  rm get-pip.py
+
+RUN echo "Install python related packages" && \
+  apt-get -y update && \
+  apt-get install -y gfortran && \
+  # numerical/algebra packages
+  apt-get install -y libblas-dev libatlas-dev liblapack-dev && \
+  # font, image for matplotlib
+  apt-get install -y libpng-dev libxft-dev && \
+  # for tkinter
+  apt-get install -y python-tk libxml2-dev libxslt-dev zlib1g-dev
+
+RUN pip --no-cache-dir install Pillow h5py ipykernel jupyter matplotlib numpy 
pandas scipy sklearn && \
+  python -m ipykernel.kernelspec
+
+# Set the locale
+# disable bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
+RUN apt-get clean && apt-get update && apt-get install -y locales
+RUN locale-gen en_US.UTF-8
+
+# Install TensorFlow CPU version.
+ENV TENSORFLOW_VERSION="1.10.0"
+RUN pip --no-cache-dir install \
+  
http://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-${TENSORFLOW_VERSION}-cp27-none-linux_x86_64.whl
+RUN apt-get update && apt-get install git -y
+
+# Install hadoop
+ENV HADOOP_VERSION="3.1.2"
+RUN wget 
http://mirrors.hust.edu.cn/apache/hadoop/common/hadoop-${HADOOP_VERSION}/hadoop-${HADOOP_VERSION}.tar.gz
 
 Review comment:
   The version number cannot be modified.
   ditto.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264623929
 
 

 ##
 File path: 
submarine/src/main/java/org/apache/zeppelin/submarine/PySubmarineInterpreter.java
 ##
 @@ -0,0 +1,84 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.submarine;
+
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
+import org.apache.zeppelin.interpreter.thrift.ParagraphInfo;
+import org.apache.zeppelin.python.IPythonInterpreter;
+import org.apache.zeppelin.python.PythonInterpreter;
+import org.apache.zeppelin.submarine.commons.SubmarineConstants;
+import org.apache.zeppelin.submarine.job.SubmarineJob;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+import java.util.Properties;
+
+public class PySubmarineInterpreter extends PythonInterpreter {
+  private static final Logger LOGGER = 
LoggerFactory.getLogger(PySubmarineInterpreter.class);
+
+  private SubmarineInterpreter submarineInterpreter = null;
+  private SubmarineContext submarineContext = null;
+
+  public PySubmarineInterpreter(Properties property) {
+super(property);
+submarineContext = SubmarineContext.getInstance();
+  }
+
+  @Override
+  public InterpreterResult interpret(String st, InterpreterContext context)
+  throws InterpreterException {
+// algorithm & checkpoint path support replaces ${username} with real user 
name
+String algorithmPath = properties.getProperty(
 
 Review comment:
   Now our users are using `python` to develop machine learning algorithms. 
Because submarine now offers online services, And have a certain amount of 
code, So I hope I can merge this phase code first. Subsequent improvements to 
ipython support.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

2019-03-12 Thread GitBox

liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop 
Submarine (Machine Learning) interpreter
URL: https://github.com/apache/zeppelin/pull/3331#discussion_r264619167
 
 

 ##
 File path: submarine/pom.xml
 ##
 @@ -0,0 +1,159 @@
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+  4.0.0
+
+  
+zeppelin-interpreter-parent
+org.apache.zeppelin
+0.9.0-SNAPSHOT
+../zeppelin-interpreter-parent/pom.xml
+  
+
+  org.apache.zeppelin
+  zeppelin-submarine
+  jar
+  0.9.0-SNAPSHOT
+  Zeppelin: Submarine interpreter
+
+  
+
+submarine
+2.7.3
 
 Review comment:
   Here Hadoop is used to upload algorithm files to HDFS, Hadoop3.1+ needs to 
be deployed locally like a spark. I have instructions in the 
`interpreter/submarine.md` documentation.
   Submarine is developing a client that does not rely on hadoop, There is only 
one JAR file package. I will update it when it is released.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

[GitHub] [zeppelin] liuxunorg commented on a change in pull request #3331: [Zeppelin-4049] Hadoop Submarine (Machine Learning) interpreter

18 matches

Site Navigation

Mail list logo

Footer information