zeppelin git commit: [HOTFIX]. Correct spark shim version
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 fd96855d9 -> 15733f4ec [HOTFIX]. Correct spark shim version ### What is this PR for? Correct the version for spark shims ### What type of PR is it? [Bug Fix ] ### Todos * [ ] - Task ### What is the Jira issue? * No jira created ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2820 from zjffdu/HotFix_Version and squashes the following commits: cadd2cc [Jeff Zhang] HOTFIX. Correct spark shim version Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/15733f4e Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/15733f4e Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/15733f4e Branch: refs/heads/branch-0.8 Commit: 15733f4ec69027d13878f7439199b16bb56afc8e Parents: fd96855 Author: Jeff Zhang Authored: Tue Feb 27 14:44:43 2018 +0800 Committer: Jeff Zhang Committed: Tue Feb 27 21:07:45 2018 +0800 -- spark/spark-shims/pom.xml | 4 ++-- spark/spark1-shims/pom.xml | 4 ++-- spark/spark2-shims/pom.xml | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/15733f4e/spark/spark-shims/pom.xml -- diff --git a/spark/spark-shims/pom.xml b/spark/spark-shims/pom.xml index 619c7a4..ca602f3 100644 --- a/spark/spark-shims/pom.xml +++ b/spark/spark-shims/pom.xml @@ -23,14 +23,14 @@ spark-parent org.apache.zeppelin -0.9.0-SNAPSHOT +0.8.0-SNAPSHOT ../pom.xml 4.0.0 org.apache.zeppelin spark-shims - 0.9.0-SNAPSHOT + 0.8.0-SNAPSHOT jar Zeppelin: Spark Shims http://git-wip-us.apache.org/repos/asf/zeppelin/blob/15733f4e/spark/spark1-shims/pom.xml -- diff --git a/spark/spark1-shims/pom.xml b/spark/spark1-shims/pom.xml index 93640c6..388c4e5 100644 --- a/spark/spark1-shims/pom.xml +++ b/spark/spark1-shims/pom.xml @@ -23,14 +23,14 @@ spark-parent org.apache.zeppelin -0.9.0-SNAPSHOT +0.8.0-SNAPSHOT ../pom.xml 4.0.0 org.apache.zeppelin spark1-shims - 0.9.0-SNAPSHOT + 0.8.0-SNAPSHOT jar Zeppelin: Spark1 Shims http://git-wip-us.apache.org/repos/asf/zeppelin/blob/15733f4e/spark/spark2-shims/pom.xml -- diff --git a/spark/spark2-shims/pom.xml b/spark/spark2-shims/pom.xml index 000e3ab..d43c35e 100644 --- a/spark/spark2-shims/pom.xml +++ b/spark/spark2-shims/pom.xml @@ -22,14 +22,14 @@ spark-parent org.apache.zeppelin -0.9.0-SNAPSHOT +0.8.0-SNAPSHOT ../pom.xml 4.0.0 org.apache.zeppelin spark2-shims - 0.9.0-SNAPSHOT + 0.8.0-SNAPSHOT jar Zeppelin: Spark2 Shims
zeppelin git commit: ZEPPELIN-3242. Listener threw an exception java.lang.NPEat o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
Repository: zeppelin
Updated Branches:
refs/heads/master 64bbba479 -> 500b74b19
ZEPPELIN-3242. Listener threw an exception java.lang.NPEat
o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
### What is this PR for?
This issue also cause spark url can not be displayed in frontend. The root
cause is that PySparkInterpreter/IPySparkInterpreter doesn't set JobGroup
correctly.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3242
### How should this be tested?
* CI pass and also manually verified.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2822 from zjffdu/ZEPPELIN-3242 and squashes the following commits:
8254162 [Jeff Zhang] ZEPPELIN-3242. Listener threw an exception java.lang.NPEat
o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/500b74b1
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/500b74b1
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/500b74b1
Branch: refs/heads/master
Commit: 500b74b196b740c810553c43216a56e23ab9caf0
Parents: 64bbba4
Author: Jeff Zhang
Authored: Tue Feb 27 20:53:54 2018 +0800
Committer: Jeff Zhang
Committed: Wed Feb 28 10:27:08 2018 +0800
--
.../zeppelin/spark/IPySparkInterpreter.java | 11 +++
.../zeppelin/spark/PySparkInterpreter.java | 13 +---
.../zeppelin/spark/IPySparkInterpreterTest.java | 33 +---
.../zeppelin/spark/NewSparkInterpreterTest.java | 17 --
.../zeppelin/spark/SparkRInterpreterTest.java | 19 +--
.../interpreter/InterpreterContext.java | 4 +++
6 files changed, 77 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/500b74b1/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
index a75fda8..3691156 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
@@ -23,6 +23,7 @@ import org.apache.zeppelin.interpreter.BaseZeppelinContext;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.LazyOpenInterpreter;
import org.apache.zeppelin.interpreter.WrappedInterpreter;
import org.apache.zeppelin.python.IPythonInterpreter;
@@ -99,6 +100,16 @@ public class IPySparkInterpreter extends IPythonInterpreter
{
}
@Override
+ public InterpreterResult interpret(String st, InterpreterContext context) {
+InterpreterContext.set(context);
+sparkInterpreter.populateSparkWebUrl(context);
+String jobGroupId = Utils.buildJobGroupId(context);
+String jobDesc = "Started by: " +
Utils.getUserName(context.getAuthenticationInfo());
+String setJobGroupStmt = "sc.setJobGroup('" + jobGroupId + "', '" +
jobDesc + "')";
+return super.interpret(setJobGroupStmt +"\n" + st, context);
+ }
+
+ @Override
public void cancel(InterpreterContext context) throws InterpreterException {
super.cancel(context);
sparkInterpreter.cancel(context);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/500b74b1/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index 0703ad7..f5e4793 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -406,16 +406,16 @@ public class PySparkInterpreter extends Interpreter
implements ExecuteResultHand
@Override
public InterpreterResult interpret(String st, InterpreterContext context)
throws InterpreterException {
+if (iPySparkInterpreter != null) {
+ return iPySparkInterpreter.interpret(st, context);
+}
+
SparkInterpreter sparkInterpreter = getS
zeppelin git commit: ZEPPELIN-3242. Listener threw an exception java.lang.NPEat o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 15733f4ec -> 18f3e547a
ZEPPELIN-3242. Listener threw an exception java.lang.NPEat
o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
### What is this PR for?
This issue also cause spark url can not be displayed in frontend. The root
cause is that PySparkInterpreter/IPySparkInterpreter doesn't set JobGroup
correctly.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3242
### How should this be tested?
* CI pass and also manually verified.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2822 from zjffdu/ZEPPELIN-3242 and squashes the following commits:
8254162 [Jeff Zhang] ZEPPELIN-3242. Listener threw an exception java.lang.NPEat
o.a.zeppelin.spark.Utils.getNoteId(Utils.java:156)
(cherry picked from commit 500b74b196b740c810553c43216a56e23ab9caf0)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/18f3e547
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/18f3e547
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/18f3e547
Branch: refs/heads/branch-0.8
Commit: 18f3e547ae8d9b3ad81782b016593512fb8664f2
Parents: 15733f4
Author: Jeff Zhang
Authored: Tue Feb 27 20:53:54 2018 +0800
Committer: Jeff Zhang
Committed: Wed Feb 28 10:27:23 2018 +0800
--
.../zeppelin/spark/IPySparkInterpreter.java | 11 +++
.../zeppelin/spark/PySparkInterpreter.java | 13 +---
.../zeppelin/spark/IPySparkInterpreterTest.java | 33 +---
.../zeppelin/spark/NewSparkInterpreterTest.java | 17 --
.../zeppelin/spark/SparkRInterpreterTest.java | 19 +--
.../interpreter/InterpreterContext.java | 4 +++
6 files changed, 77 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/18f3e547/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
index a75fda8..3691156 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/IPySparkInterpreter.java
@@ -23,6 +23,7 @@ import org.apache.zeppelin.interpreter.BaseZeppelinContext;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
+import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.LazyOpenInterpreter;
import org.apache.zeppelin.interpreter.WrappedInterpreter;
import org.apache.zeppelin.python.IPythonInterpreter;
@@ -99,6 +100,16 @@ public class IPySparkInterpreter extends IPythonInterpreter
{
}
@Override
+ public InterpreterResult interpret(String st, InterpreterContext context) {
+InterpreterContext.set(context);
+sparkInterpreter.populateSparkWebUrl(context);
+String jobGroupId = Utils.buildJobGroupId(context);
+String jobDesc = "Started by: " +
Utils.getUserName(context.getAuthenticationInfo());
+String setJobGroupStmt = "sc.setJobGroup('" + jobGroupId + "', '" +
jobDesc + "')";
+return super.interpret(setJobGroupStmt +"\n" + st, context);
+ }
+
+ @Override
public void cancel(InterpreterContext context) throws InterpreterException {
super.cancel(context);
sparkInterpreter.cancel(context);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/18f3e547/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index 0703ad7..f5e4793 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -406,16 +406,16 @@ public class PySparkInterpreter extends Interpreter
implements ExecuteResultHand
@Override
public InterpreterResult interpret(String st, InterpreterContext context)
throws InterpreterException {
+if (iPySparkInterpreter != null) {
+ ret
zeppelin git commit: ZEPPELIN-3154. Fixed Checkstyle errors and warnings in flink module
Repository: zeppelin
Updated Branches:
refs/heads/master 500b74b19 -> 65b797c22
ZEPPELIN-3154. Fixed Checkstyle errors and warnings in flink module
### What is this PR for?
Fixed the Checkstyle errors and warnings in the flink module.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3154
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Jan Hentschel
Closes #2795 from HorizonNet/ZEPPELIN-3154 and squashes the following commits:
f70e93d [Jan Hentschel] ZEPPELIN-3154. Fixed Checkstyle errors and warnings in
flink module
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/65b797c2
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/65b797c2
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/65b797c2
Branch: refs/heads/master
Commit: 65b797c22eadebb9a18575e91fdcfe33722762c8
Parents: 500b74b
Author: Jan Hentschel
Authored: Tue Feb 13 12:21:21 2018 +0100
Committer: Jeff Zhang
Committed: Wed Feb 28 11:29:31 2018 +0800
--
flink/pom.xml | 253 ++-
.../apache/zeppelin/flink/FlinkInterpreter.java | 37 +--
.../zeppelin/flink/FlinkInterpreterTest.java| 22 +-
3 files changed, 164 insertions(+), 148 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/65b797c2/flink/pom.xml
--
diff --git a/flink/pom.xml b/flink/pom.xml
index 455092d..70c076d 100644
--- a/flink/pom.xml
+++ b/flink/pom.xml
@@ -154,142 +154,149 @@
- net.alchim31.maven
- scala-maven-plugin
- ${plugin.scalamaven.version}
-
-
-
- scala-compile-first
- process-resources
-
- compile
-
-
-
-
-
- scala-test-compile
- process-test-resources
-
- testCompile
-
-
-
-
-
- -Xms128m
- -Xmx512m
-
-
-
- org.scalamacros
- paradise_${scala.version}
- ${scala.macros.version}
-
-
-
+net.alchim31.maven
+scala-maven-plugin
+${plugin.scalamaven.version}
+
+
+
+scala-compile-first
+process-resources
+
+ compile
+
+
+
+
+
+scala-test-compile
+process-test-resources
+
+ testCompile
+
+
+
+
+
+-Xms128m
+-Xmx512m
+
+
+
+ org.scalamacros
+ paradise_${scala.version}
+ ${scala.macros.version}
+
+
+
- org.apache.maven.plugins
- maven-eclipse-plugin
- ${plugin.eclipse.version}
-
- true
-
- org.scala-ide.sdt.core.scalanature
- org.eclipse.jdt.core.javanature
-
-
- org.scala-ide.sdt.core.scalabuilder
-
-
-
org.scala-ide.sdt.launching.SCALA_CONTAINER
-
org.eclipse.jdt.launching.JRE_CONTAINER
-
-
-
- **/*.scala
- **/*.java
-
-
+org.apache.maven.plugins
+maven-eclipse-plugin
+${plugin.eclipse.version}
+
+ true
+
+org.scala-ide.sdt.core.scalanature
+org.eclipse.jdt.core.javanature
+
+
+org.scala-ide.sdt.core.scalabuilder
+
+
+
org.scala-ide.sdt.launching.SCALA_CONTAINER
+
org.eclipse.jdt.launching.JRE_CONTAINER
+
+
+
+**/*.scala
+**/*.java
+
+
- org.codehaus.mojo
- build-helper-maven-plugin
- ${plugin.buildhelper.version}
-
-
-
- add-source
- generate-sources
-
- add-source
-
-
-
- src/main/scala
-
-
-
-
-
- add-test-source
- generate-test-sources
-
- add-test-source
-
-
-
- src/test/scala
-
-
-
zeppelin git commit: ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
Repository: zeppelin
Updated Branches:
refs/heads/master 2c322d72b -> fa9f88ba4
ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
### What is this PR for?
The root cause is the sparkr package is not distributed correctly. This PR
correct the distributed file name and also add unit test for this.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3273
### How should this be tested?
* Unit test added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2823 from zjffdu/ZEPPELIN-3273 and squashes the following commits:
7f6c3db [Jeff Zhang] ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/fa9f88ba
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/fa9f88ba
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/fa9f88ba
Branch: refs/heads/master
Commit: fa9f88ba4fbd8f8faa0ad7cf762cfaf795eb0e51
Parents: 2c322d7
Author: Jeff Zhang
Authored: Wed Feb 28 16:55:52 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 2 15:06:15 2018 +0800
--
.travis.yml | 4 +--
.../zeppelin/spark/IPySparkInterpreterTest.java | 28 +++-
.../launcher/SparkInterpreterLauncher.java | 3 ++-
.../interpreter/SparkInterpreterModeTest.java | 16 +++
4 files changed, 35 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/fa9f88ba/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index c31694a..bc568fc 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,9 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- addons:
-firefox: "31.0"
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples -Pscala-2.11"
BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/fa9f88ba/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
--
diff --git
a/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
b/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
index 46a3a72..17c2af8 100644
---
a/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
+++
b/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
@@ -103,6 +103,7 @@ public class IPySparkInterpreterTest {
InterpreterResult result = iPySparkInterpreter.interpret("sc.version",
context);
Thread.sleep(100);
assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+String sparkVersion =
context.out.getInterpreterResultMessages().get(0).getData();
// spark url is sent
verify(mockRemoteEventClient).onMetaInfosReceived(any(Map.class));
@@ -117,18 +118,17 @@ public class IPySparkInterpreterTest {
// spark sql
context = getInterpreterContext();
-if (interpreterResultMessages.get(0).getData().startsWith("'1.") ||
-interpreterResultMessages.get(0).getData().startsWith("u'1.")) {
+if (!isSpark2(sparkVersion)) {
result = iPySparkInterpreter.interpret("df =
sqlContext.createDataFrame([(1,'a'),(2,'b')])\ndf.show()", context);
assertEquals(InterpreterResult.Code.SU
zeppelin git commit: ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 d958b568c -> 7187932a9
ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
### What is this PR for?
The root cause is the sparkr package is not distributed correctly. This PR
correct the distributed file name and also add unit test for this.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3273
### How should this be tested?
* Unit test added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2823 from zjffdu/ZEPPELIN-3273 and squashes the following commits:
7f6c3db [Jeff Zhang] ZEPPELIN-3273. SparkRInterpreter doesn't work in yarn mode
(cherry picked from commit fa9f88ba4fbd8f8faa0ad7cf762cfaf795eb0e51)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7187932a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7187932a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7187932a
Branch: refs/heads/branch-0.8
Commit: 7187932a9662477cb9446eaf9c5e9e6e97db77b5
Parents: d958b56
Author: Jeff Zhang
Authored: Wed Feb 28 16:55:52 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 2 15:06:29 2018 +0800
--
.travis.yml | 4 +--
.../zeppelin/spark/IPySparkInterpreterTest.java | 28 +++-
.../launcher/SparkInterpreterLauncher.java | 3 ++-
.../interpreter/SparkInterpreterModeTest.java | 16 +++
4 files changed, 35 insertions(+), 16 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7187932a/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index c31694a..bc568fc 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,9 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- addons:
-firefox: "31.0"
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples -Pscala-2.11"
BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7187932a/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
--
diff --git
a/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
b/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
index 46a3a72..17c2af8 100644
---
a/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
+++
b/spark/interpreter/src/test/java/org/apache/zeppelin/spark/IPySparkInterpreterTest.java
@@ -103,6 +103,7 @@ public class IPySparkInterpreterTest {
InterpreterResult result = iPySparkInterpreter.interpret("sc.version",
context);
Thread.sleep(100);
assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+String sparkVersion =
context.out.getInterpreterResultMessages().get(0).getData();
// spark url is sent
verify(mockRemoteEventClient).onMetaInfosReceived(any(Map.class));
@@ -117,18 +118,17 @@ public class IPySparkInterpreterTest {
// spark sql
context = getInterpreterContext();
-if (interpreterResultMessages.get(0).getData().startsWith("'1.") ||
-interpreterResultMessages.get(0).getData().startsWith("u'1.")) {
+if (!isSpark2(sparkVersion)) {
result = iPySparkInterpreter.interpret("df =
sqlContext.createDataFrame([(1,'a'
zeppelin git commit: ZEPPELIN-3278. Avoid duplicated interpreter setting
Repository: zeppelin
Updated Branches:
refs/heads/master fa9f88ba4 -> 21dc20d88
ZEPPELIN-3278. Avoid duplicated interpreter setting
### What is this PR for?
Straightforward bugfix for ZEPPELIN-3278. Just don't create interpreter setting
instance when registering from template.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3278
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2824 from zjffdu/ZEPPELIN-3278 and squashes the following commits:
6ece9ca [Jeff Zhang] [ZEPPELIN-3278] Avoid duplicated interpreter setting
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/21dc20d8
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/21dc20d8
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/21dc20d8
Branch: refs/heads/master
Commit: 21dc20d8864915475f58705d8bc2fc8678c22dbe
Parents: fa9f88b
Author: Jeff Zhang
Authored: Fri Mar 2 09:47:56 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 2 15:09:04 2018 +0800
--
.../zeppelin/interpreter/InterpreterSettingManager.java | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/21dc20d8/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 0601c6f..711812e 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -192,13 +192,14 @@ public class InterpreterSettingManager {
}
/**
- * Load interpreter setting from interpreter-setting.json
+ * Load interpreter setting from interpreter.json
*/
private void loadFromFile() throws IOException {
InterpreterInfoSaving infoSaving =
configStorage.loadInterpreterSettings();
if (infoSaving == null) {
- // nothing to read
+ // it is fresh zeppelin instance if there's no interpreter.json, just
create interpreter
+ // setting from interpreterSettingTemplates
for (InterpreterSetting interpreterSettingTemplate :
interpreterSettingTemplates.values()) {
InterpreterSetting interpreterSetting = new
InterpreterSetting(interpreterSettingTemplate);
initInterpreterSetting(interpreterSetting);
@@ -403,14 +404,10 @@ public class InterpreterSettingManager {
.setIntepreterSettingManager(this)
.create();
-LOGGER.info("Register InterpreterSettingTemplate & Create
InterpreterSetting: {}",
+LOGGER.info("Register InterpreterSettingTemplate: {}",
interpreterSettingTemplate.getName());
interpreterSettingTemplates.put(interpreterSettingTemplate.getName(),
interpreterSettingTemplate);
-
-InterpreterSetting interpreterSetting = new
InterpreterSetting(interpreterSettingTemplate);
-initInterpreterSetting(interpreterSetting);
-interpreterSettings.put(interpreterSetting.getName(), interpreterSetting);
}
@VisibleForTesting
zeppelin git commit: ZEPPELIN-3278. Avoid duplicated interpreter setting
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 7187932a9 -> 3078cd47a
ZEPPELIN-3278. Avoid duplicated interpreter setting
### What is this PR for?
Straightforward bugfix for ZEPPELIN-3278. Just don't create interpreter setting
instance when registering from template.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3278
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2824 from zjffdu/ZEPPELIN-3278 and squashes the following commits:
6ece9ca [Jeff Zhang] [ZEPPELIN-3278] Avoid duplicated interpreter setting
(cherry picked from commit 21dc20d8864915475f58705d8bc2fc8678c22dbe)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/3078cd47
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/3078cd47
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/3078cd47
Branch: refs/heads/branch-0.8
Commit: 3078cd47a39be2c0e4afa9229b6399e19fcae583
Parents: 7187932
Author: Jeff Zhang
Authored: Fri Mar 2 09:47:56 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 2 15:09:18 2018 +0800
--
.../zeppelin/interpreter/InterpreterSettingManager.java | 11 ---
1 file changed, 4 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3078cd47/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 0601c6f..711812e 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -192,13 +192,14 @@ public class InterpreterSettingManager {
}
/**
- * Load interpreter setting from interpreter-setting.json
+ * Load interpreter setting from interpreter.json
*/
private void loadFromFile() throws IOException {
InterpreterInfoSaving infoSaving =
configStorage.loadInterpreterSettings();
if (infoSaving == null) {
- // nothing to read
+ // it is fresh zeppelin instance if there's no interpreter.json, just
create interpreter
+ // setting from interpreterSettingTemplates
for (InterpreterSetting interpreterSettingTemplate :
interpreterSettingTemplates.values()) {
InterpreterSetting interpreterSetting = new
InterpreterSetting(interpreterSettingTemplate);
initInterpreterSetting(interpreterSetting);
@@ -403,14 +404,10 @@ public class InterpreterSettingManager {
.setIntepreterSettingManager(this)
.create();
-LOGGER.info("Register InterpreterSettingTemplate & Create
InterpreterSetting: {}",
+LOGGER.info("Register InterpreterSettingTemplate: {}",
interpreterSettingTemplate.getName());
interpreterSettingTemplates.put(interpreterSettingTemplate.getName(),
interpreterSettingTemplate);
-
-InterpreterSetting interpreterSetting = new
InterpreterSetting(interpreterSettingTemplate);
-initInterpreterSetting(interpreterSetting);
-interpreterSettings.put(interpreterSetting.getName(), interpreterSetting);
}
@VisibleForTesting
zeppelin git commit: [HOTFIX] Fix branch-0.8 build
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 3078cd47a -> f646dc385
[HOTFIX] Fix branch-0.8 build
### What is this PR for?
This is a hotfix PR for branch-0.8 build
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* No jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2830 from zjffdu/Hotfix_0.8 and squashes the following commits:
59c2af4 [Jeff Zhang] [HOTFIX] Fix branch-0.8 build
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/f646dc38
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/f646dc38
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/f646dc38
Branch: refs/heads/branch-0.8
Commit: f646dc385aeee354cedb438b6100654aff9abb22
Parents: 3078cd4
Author: Jeff Zhang
Authored: Tue Feb 27 21:23:36 2018 +0800
Committer: Jeff Zhang
Committed: Sat Mar 3 14:33:38 2018 +0800
--
.travis.yml | 14 -
.../zeppelin/livy/SessionDeadException.java | 3 ++
.../zeppelin/spark/IPySparkInterpreter.java | 2 +-
.../org/apache/zeppelin/spark/Spark1Shims.java | 3 ++
.../org/apache/zeppelin/spark/Spark2Shims.java | 3 ++
zeppelin-zengine/pom.xml| 33
.../org/apache/zeppelin/notebook/Notebook.java | 6 ++--
7 files changed, 21 insertions(+), 43 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/f646dc38/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index bc568fc..3e57ffa 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -73,37 +73,37 @@ matrix:
dist: trusty
addons:
firefox: "31.0"
- env: PYTHON="2" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6"
PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Phelium-dev -Pexamples
-Pintegration" BUILD_FLAG="install -DskipTests -DskipRat" TEST_FLAG="verify
-DskipRat" TEST_PROJECTS="-pl .,zeppelin-integration -DfailIfNoTests=false"
+ env: PYTHON="2" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6"
PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Phelium-dev -Pexamples
-Pintegration" BUILD_FLAG="package -DskipTests -DskipRat" TEST_FLAG="verify
-DskipRat" TEST_PROJECTS="-pl .,zeppelin-integration -DfailIfNoTests=false"
# Test interpreter modules
- jdk: "openjdk7"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.10" PROFILE="-Pscalding"
BUILD_FLAG="install -DskipTests -DskipRat -Pr" TEST_FLAG="test -DskipRat"
MODULES="-pl $(echo .,zeppelin-interpreter,${INTERPRETERS} | sed 's/!//g')"
TEST_PROJECTS=""
+ env: PYTHON="3" SCALA_VER="2.10" PROFILE="-Pscalding"
BUILD_FLAG="package -DskipTests -DskipRat -Pr" TEST_FLAG="test -DskipRat"
MODULES="-pl $(echo .,zeppelin-interpreter,${INTERPRETERS} | sed 's/!//g')"
TEST_PROJECTS=""
# Test spark module for 2.2.0 with scala 2.11
- jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.2 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true"
BUILD_FLAG="install -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am"
MODULES="-pl zeppelin-server,spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest,org.apache.zeppelin.spark.*
-DfailIfNoTests=false"
+ env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.2 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true"
BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am"
MODULES="-pl zeppelin-server,spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest,org.apache.zeppelin.spark.*
-DfailIfNoTests=false"
# Test spark module for 2.1.0 with scala 2.11
- jdk: "openjdk7"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.1.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.1 -Phadoop2 -Phadoop-2.6 -Pscala-2.11" SP
zeppelin git commit: ZEPPELIN-3281. Apply getRelativePath when it is LocalConfigStorage
Repository: zeppelin
Updated Branches:
refs/heads/master 21dc20d88 -> bfc93dc03
ZEPPELIN-3281. Apply getRelativePath when it is LocalConfigStorage
### What is this PR for?
When it is LocalConfigStorage, we should use getRelativePath to get the config
path.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3281
### How should this be tested?
* Unit test added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2831 from zjffdu/ZEPPELIN-3281 and squashes the following commits:
3438577 [Jeff Zhang] ZEPPELIN-3281. Apply getRelativePath when it is
LocalConfigStorage
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/bfc93dc0
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/bfc93dc0
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/bfc93dc0
Branch: refs/heads/master
Commit: bfc93dc030f94be0ac624fa8d58279d553e3ff07
Parents: 21dc20d
Author: Jeff Zhang
Authored: Fri Mar 2 18:57:10 2018 +0800
Committer: Jeff Zhang
Committed: Sat Mar 3 14:42:16 2018 +0800
--
.../org/apache/zeppelin/conf/ZeppelinConfiguration.java | 8 +++-
.../org/apache/zeppelin/storage/LocalConfigStorage.java | 1 +
.../apache/zeppelin/conf/ZeppelinConfigurationTest.java | 12
3 files changed, 20 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bfc93dc0/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index e0ebfa2..81f9341 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -524,7 +524,13 @@ public class ZeppelinConfiguration extends
XMLConfiguration {
"conf directory " + ConfVars.ZEPPELIN_CONF_DIR.varName);
return getConfDir();
}
-return fsConfigDir;
+if (getString(ConfVars.ZEPPELIN_CONFIG_STORAGE_CLASS)
+.equals("org.apache.zeppelin.storage.LocalConfigStorage")) {
+ // only apply getRelativeDir when it is LocalConfigStorage
+ return getRelativeDir(fsConfigDir);
+} else {
+ return fsConfigDir;
+}
}
public List getAllowedOrigins()
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bfc93dc0/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
index c1edbb5..464d6ce 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
@@ -50,6 +50,7 @@ public class LocalConfigStorage extends ConfigStorage {
@Override
public void save(InterpreterInfoSaving settingInfos) throws IOException {
+LOGGER.info("Save Interpreter Setting to " +
interpreterSettingPath.getAbsolutePath());
writeToFile(settingInfos.toJson(), interpreterSettingPath);
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bfc93dc0/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
index 2b50427..1771fd3 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
@@ -22,6 +22,7 @@ import
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
import org.junit.Before;
import org.junit.Test;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.net.MalformedURLException;
@@ -100,4 +101,15 @@ public class ZeppelinConfigurationTest {
Assert.assertEquals("/usr/lib/zeppelin", conf.getZeppelinHome());
Assert.assertEquals("/usr/lib/zeppelin/conf", conf.getConfDir());
zeppelin git commit: ZEPPELIN-3281. Apply getRelativePath when it is LocalConfigStorage
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 f646dc385 -> 3fd2fec0d
ZEPPELIN-3281. Apply getRelativePath when it is LocalConfigStorage
### What is this PR for?
When it is LocalConfigStorage, we should use getRelativePath to get the config
path.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3281
### How should this be tested?
* Unit test added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2831 from zjffdu/ZEPPELIN-3281 and squashes the following commits:
3438577 [Jeff Zhang] ZEPPELIN-3281. Apply getRelativePath when it is
LocalConfigStorage
(cherry picked from commit bfc93dc030f94be0ac624fa8d58279d553e3ff07)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/3fd2fec0
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/3fd2fec0
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/3fd2fec0
Branch: refs/heads/branch-0.8
Commit: 3fd2fec0d7642d7e585110ce7510bf2eedaba721
Parents: f646dc3
Author: Jeff Zhang
Authored: Fri Mar 2 18:57:10 2018 +0800
Committer: Jeff Zhang
Committed: Sat Mar 3 14:42:27 2018 +0800
--
.../org/apache/zeppelin/conf/ZeppelinConfiguration.java | 8 +++-
.../org/apache/zeppelin/storage/LocalConfigStorage.java | 1 +
.../apache/zeppelin/conf/ZeppelinConfigurationTest.java | 12
3 files changed, 20 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3fd2fec0/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
index dd5491d..725d94b 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java
@@ -524,7 +524,13 @@ public class ZeppelinConfiguration extends
XMLConfiguration {
"conf directory " + ConfVars.ZEPPELIN_CONF_DIR.varName);
return getConfDir();
}
-return fsConfigDir;
+if (getString(ConfVars.ZEPPELIN_CONFIG_STORAGE_CLASS)
+.equals("org.apache.zeppelin.storage.LocalConfigStorage")) {
+ // only apply getRelativeDir when it is LocalConfigStorage
+ return getRelativeDir(fsConfigDir);
+} else {
+ return fsConfigDir;
+}
}
public List getAllowedOrigins()
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3fd2fec0/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
index c1edbb5..464d6ce 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/storage/LocalConfigStorage.java
@@ -50,6 +50,7 @@ public class LocalConfigStorage extends ConfigStorage {
@Override
public void save(InterpreterInfoSaving settingInfos) throws IOException {
+LOGGER.info("Save Interpreter Setting to " +
interpreterSettingPath.getAbsolutePath());
writeToFile(settingInfos.toJson(), interpreterSettingPath);
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3fd2fec0/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
index 2b50427..1771fd3 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/conf/ZeppelinConfigurationTest.java
@@ -22,6 +22,7 @@ import
org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
import org.junit.Before;
import org.junit.Test;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import java.net.MalformedURLException;
@@ -100,4 +101,15 @@ public class ZeppelinConfigurationTest {
Assert.assertEquals("/u
[1/2] zeppelin git commit: ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of spark
Repository: zeppelin
Updated Branches:
refs/heads/master d90716d4d -> d6cdd56d8
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/d6cdd56d/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
new file mode 100644
index 000..50930a7
--- /dev/null
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
@@ -0,0 +1,190 @@
+package org.apache.zeppelin.interpreter;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+public class SparkIntegrationTest {
+ private static Logger LOGGER =
LoggerFactory.getLogger(SparkIntegrationTest.class);
+
+ private static MiniHadoopCluster hadoopCluster;
+ private static MiniZeppelin zeppelin;
+ private static InterpreterFactory interpreterFactory;
+ private static InterpreterSettingManager interpreterSettingManager;
+
+ private String sparkVersion;
+ private String sparkHome;
+
+ public SparkIntegrationTest(String sparkVersion) {
+LOGGER.info("Testing SparkVersion: " + sparkVersion);
+this.sparkVersion = sparkVersion;
+this.sparkHome = SparkDownloadUtils.downloadSpark(sparkVersion);
+ }
+
+ @Parameterized.Parameters
+ public static List data() {
+return Arrays.asList(new Object[][]{
+{"2.2.1"},
+{"2.1.2"},
+{"2.0.2"},
+{"1.6.3"}
+});
+ }
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+hadoopCluster = new MiniHadoopCluster();
+hadoopCluster.start();
+
+zeppelin = new MiniZeppelin();
+zeppelin.start();
+interpreterFactory = zeppelin.getInterpreterFactory();
+interpreterSettingManager = zeppelin.getInterpreterSettingManager();
+ }
+
+ @AfterClass
+ public static void tearDown() throws IOException {
+if (zeppelin != null) {
+ zeppelin.stop();
+}
+if (hadoopCluster != null) {
+ hadoopCluster.stop();
+}
+ }
+
+ private void testInterpreterBasics() throws IOException,
InterpreterException {
+// test SparkInterpreter
+interpreterSettingManager.setInterpreterBinding("user1", "note1",
interpreterSettingManager.getInterpreterSettingIds());
+Interpreter sparkInterpreter = interpreterFactory.getInterpreter("user1",
"note1", "spark.spark");
+
+InterpreterContext context = new
InterpreterContext.Builder().setNoteId("note1").setParagraphId("paragraph_1").getContext();
+InterpreterResult interpreterResult =
sparkInterpreter.interpret("sc.version", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+String detectedSparkVersion = interpreterResult.message().get(0).getData();
+assertTrue(detectedSparkVersion +" doesn't contain " + this.sparkVersion,
detectedSparkVersion.contains(this.sparkVersion));
+interpreterResult = sparkInterpreter.interpret("sc.range(1,10).sum()",
context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+assertTrue(interpreterResult.msg.get(0).getData().contains("45"));
+
+// test PySparkInterpreter
+Interpreter pySparkInterpreter =
interpreterFactory.getInterpreter("user1", "note1", "spark.pyspark");
+interpreterResult =
pySparkInterpreter.interpret("sqlContext.createDataFrame([(1,'a'),(2,'b')],
['id','name']).registerTempTable('test')", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+
+// test IPySparkInterpreter
+Interpreter ipySparkInterpreter =
interpreterFactory.getInterpreter("user1", "note1", "spark.ipyspark");
+interpreterResult =
ipySparkInterpreter.interpret("sqlContext.table('test').show()", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+
+// test SparkSQLInterpreter
+Interpreter sqlInterpreter = interpreterFactory.getInterpreter("user1",
"note1", "spark.sql");
+interpreterResult = sqlInterpreter.interpret("select count(1) as c from
test", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+assertEquals(Interpr
[2/2] zeppelin git commit: ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of spark
ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of
spark
### What is this PR for?
This PR is trying to add test for one build of zeppelin to work with multiple
versions of spark. There's 2 main system test
* `SparkIntegrationTest` ( It launches spark interpreter from interpreter
setting component in 3 modes: local, yarn-client, yarn-cluster)
* `ZeppelinSparkClusterTest` (It launch spark interpreter in local mode from
Paragraph side
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3254
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2832 from zjffdu/ZEPPELIN-3254 and squashes the following commits:
3b06ed9 [Jeff Zhang] ZEPPELIN-3254. Enable one build of zeppelin to work with
multiple versions of spark
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/d6cdd56d
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/d6cdd56d
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/d6cdd56d
Branch: refs/heads/master
Commit: d6cdd56d8fbc551dee7c334eb403c19bc65d6e1f
Parents: d90716d
Author: Jeff Zhang
Authored: Thu Feb 22 21:05:23 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 09:57:18 2018 +0800
--
.travis.yml | 30 +-
.../zeppelin/spark/SparkRInterpreter.java |8 +-
.../interpreter/InterpreterProperty.java|1 +
zeppelin-server/notebook/.python.recovery.crc | Bin 12 -> 0 bytes
zeppelin-server/notebook/python.recovery|1 -
zeppelin-server/pom.xml |9 +
.../zeppelin/rest/AbstractTestRestApi.java | 211 +---
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 1140 +-
.../src/test/resources/log4j.properties |3 +-
zeppelin-zengine/pom.xml| 14 +
.../interpreter/SparkDownloadUtils.java | 91 ++
.../interpreter/SparkIntegrationTest.java | 190 +++
.../interpreter/SparkInterpreterModeTest.java | 163 ---
.../src/test/resources/log4j.properties |6 +-
14 files changed, 984 insertions(+), 883 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/d6cdd56d/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index bc568fc..aba2944 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,7 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
@@ -78,32 +78,46 @@ matrix:
# Test interpreter modules
- jdk: "openjdk7"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.10" PROFILE="-Pscalding"
BUILD_FLAG="install -DskipTests -DskipRat -Pr" TEST_FLAG="test -DskipRat"
MODULES="-pl $(echo .,zeppelin-interpreter,${INTERPRETERS} | sed 's/!//g')"
TEST_PROJECTS=""
+ env: PYTHON="3" SCALA_VER="2.10" PROFILE="-Pscalding"
BUILD_FLAG="package -DskipTests -DskipRat -Pr" TEST_FLAG="test -DskipRat"
MODULES="-pl $(echo .,zeppelin-interpreter,${INTERPRETERS} | sed 's/!//g')"
TEST_PROJECTS=""
+
+# Run ZeppelinSparkClusterTest & SparkIntegrationTest in one build would
exceed the tim
[1/2] zeppelin git commit: ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of spark
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 f1c2b5b45 -> 226e5b8b6
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/226e5b8b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
new file mode 100644
index 000..50930a7
--- /dev/null
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/SparkIntegrationTest.java
@@ -0,0 +1,190 @@
+package org.apache.zeppelin.interpreter;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationsResponse;
+import org.apache.hadoop.yarn.api.records.YarnApplicationState;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.EnumSet;
+import java.util.List;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+
+@RunWith(value = Parameterized.class)
+public class SparkIntegrationTest {
+ private static Logger LOGGER =
LoggerFactory.getLogger(SparkIntegrationTest.class);
+
+ private static MiniHadoopCluster hadoopCluster;
+ private static MiniZeppelin zeppelin;
+ private static InterpreterFactory interpreterFactory;
+ private static InterpreterSettingManager interpreterSettingManager;
+
+ private String sparkVersion;
+ private String sparkHome;
+
+ public SparkIntegrationTest(String sparkVersion) {
+LOGGER.info("Testing SparkVersion: " + sparkVersion);
+this.sparkVersion = sparkVersion;
+this.sparkHome = SparkDownloadUtils.downloadSpark(sparkVersion);
+ }
+
+ @Parameterized.Parameters
+ public static List data() {
+return Arrays.asList(new Object[][]{
+{"2.2.1"},
+{"2.1.2"},
+{"2.0.2"},
+{"1.6.3"}
+});
+ }
+
+ @BeforeClass
+ public static void setUp() throws IOException {
+hadoopCluster = new MiniHadoopCluster();
+hadoopCluster.start();
+
+zeppelin = new MiniZeppelin();
+zeppelin.start();
+interpreterFactory = zeppelin.getInterpreterFactory();
+interpreterSettingManager = zeppelin.getInterpreterSettingManager();
+ }
+
+ @AfterClass
+ public static void tearDown() throws IOException {
+if (zeppelin != null) {
+ zeppelin.stop();
+}
+if (hadoopCluster != null) {
+ hadoopCluster.stop();
+}
+ }
+
+ private void testInterpreterBasics() throws IOException,
InterpreterException {
+// test SparkInterpreter
+interpreterSettingManager.setInterpreterBinding("user1", "note1",
interpreterSettingManager.getInterpreterSettingIds());
+Interpreter sparkInterpreter = interpreterFactory.getInterpreter("user1",
"note1", "spark.spark");
+
+InterpreterContext context = new
InterpreterContext.Builder().setNoteId("note1").setParagraphId("paragraph_1").getContext();
+InterpreterResult interpreterResult =
sparkInterpreter.interpret("sc.version", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+String detectedSparkVersion = interpreterResult.message().get(0).getData();
+assertTrue(detectedSparkVersion +" doesn't contain " + this.sparkVersion,
detectedSparkVersion.contains(this.sparkVersion));
+interpreterResult = sparkInterpreter.interpret("sc.range(1,10).sum()",
context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+assertTrue(interpreterResult.msg.get(0).getData().contains("45"));
+
+// test PySparkInterpreter
+Interpreter pySparkInterpreter =
interpreterFactory.getInterpreter("user1", "note1", "spark.pyspark");
+interpreterResult =
pySparkInterpreter.interpret("sqlContext.createDataFrame([(1,'a'),(2,'b')],
['id','name']).registerTempTable('test')", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+
+// test IPySparkInterpreter
+Interpreter ipySparkInterpreter =
interpreterFactory.getInterpreter("user1", "note1", "spark.ipyspark");
+interpreterResult =
ipySparkInterpreter.interpret("sqlContext.table('test').show()", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+
+// test SparkSQLInterpreter
+Interpreter sqlInterpreter = interpreterFactory.getInterpreter("user1",
"note1", "spark.sql");
+interpreterResult = sqlInterpreter.interpret("select count(1) as c from
test", context);
+assertEquals(InterpreterResult.Code.SUCCESS, interpreterResult.code);
+assertEquals(Int
[2/2] zeppelin git commit: ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of spark
ZEPPELIN-3254. Enable one build of zeppelin to work with multiple versions of
spark
This PR is trying to add test for one build of zeppelin to work with multiple
versions of spark. There's 2 main system test
* `SparkIntegrationTest` ( It launches spark interpreter from interpreter
setting component in 3 modes: local, yarn-client, yarn-cluster)
* `ZeppelinSparkClusterTest` (It launch spark interpreter in local mode from
Paragraph side
[Improvement]
* [ ] - Task
* https://issues.apache.org/jira/browse/ZEPPELIN-3254
* CI pass
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2832 from zjffdu/ZEPPELIN-3254 and squashes the following commits:
3b06ed9 [Jeff Zhang] ZEPPELIN-3254. Enable one build of zeppelin to work with
multiple versions of spark
(cherry picked from commit d6cdd56d8fbc551dee7c334eb403c19bc65d6e1f)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/226e5b8b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/226e5b8b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/226e5b8b
Branch: refs/heads/branch-0.8
Commit: 226e5b8b691bf56cc907cd0b85b5117769064c9b
Parents: f1c2b5b
Author: Jeff Zhang
Authored: Thu Feb 22 21:05:23 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 10:01:13 2018 +0800
--
.travis.yml | 28 +-
.../zeppelin/spark/SparkRInterpreter.java |8 +-
.../interpreter/InterpreterProperty.java|1 +
zeppelin-server/notebook/.python.recovery.crc | Bin 12 -> 0 bytes
zeppelin-server/notebook/python.recovery|1 -
zeppelin-server/pom.xml |9 +
.../zeppelin/rest/AbstractTestRestApi.java | 211 +---
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 1140 +-
.../src/test/resources/log4j.properties |3 +-
zeppelin-zengine/pom.xml| 14 +
.../interpreter/SparkDownloadUtils.java | 91 ++
.../interpreter/SparkIntegrationTest.java | 190 +++
.../interpreter/SparkInterpreterModeTest.java | 163 ---
.../src/test/resources/log4j.properties |6 +-
14 files changed, 983 insertions(+), 882 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/226e5b8b/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 3e57ffa..df96c38 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -66,7 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
@@ -80,30 +80,44 @@ matrix:
dist: trusty
env: PYTHON="3" SCALA_VER="2.10" PROFILE="-Pscalding"
BUILD_FLAG="package -DskipTests -DskipRat -Pr" TEST_FLAG="test -DskipRat"
MODULES="-pl $(echo .,zeppelin-interpreter,${INTERPRETERS} | sed 's/!//g')"
TEST_PROJECTS=""
+# Run ZeppelinSparkClusterTest & SparkIntegrationTest in one build would
exceed the time limitation of travis, so running them separately
+
+# Test spark interpreter with different spark versions under python2, only
run ZeppelinSparkClusterTest
+- sudo: required
+ jdk: "oraclejdk8"
+ dist: trusty
+ env: PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package
-DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-p
zeppelin git commit: ZEPPELIN-3291. Throw exception instead of return null for InterpreterFactory.getInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master d6cdd56d8 -> e148c8cff
ZEPPELIN-3291. Throw exception instead of return null for
InterpreterFactory.getInterpreter
### What is this PR for?
Minor code refactoring to throw exception when no interpreter is found
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3291
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2836 from zjffdu/ZEPPELIN-3291 and squashes the following commits:
bee319c [Jeff Zhang] ZEPPELIN-3291. Throw exception instead of return null for
InterpreterFactory.getInterpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/e148c8cf
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/e148c8cf
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/e148c8cf
Branch: refs/heads/master
Commit: e148c8cfff04a7a299b2d626759a662a3f91087b
Parents: d6cdd56
Author: Jeff Zhang
Authored: Mon Mar 5 12:58:19 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 10:08:23 2018 +0800
--
.../zeppelin/helium/ApplicationException.java | 4 ++
.../apache/zeppelin/socket/NotebookServer.java | 7 ++-
.../java/org/apache/zeppelin/helium/Helium.java | 7 ++-
.../helium/HeliumApplicationFactory.java| 24 ++---
.../interpreter/InterpreterFactory.java | 20 +++
.../InterpreterNotFoundException.java | 22
.../org/apache/zeppelin/notebook/Paragraph.java | 56 +++-
.../helium/HeliumApplicationFactoryTest.java| 11 +++-
.../interpreter/InterpreterFactoryTest.java | 12 +++--
.../InterpreterSettingManagerTest.java | 8 +--
.../org/apache/zeppelin/notebook/NoteTest.java | 11 ++--
.../apache/zeppelin/notebook/NotebookTest.java | 12 +++--
12 files changed, 133 insertions(+), 61 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e148c8cf/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
index 4bf0ac2..d3c6488 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
@@ -31,4 +31,8 @@ public class ApplicationException extends Exception {
public ApplicationException() {
}
+
+ public ApplicationException(String message, Throwable cause) {
+super(message, cause);
+ }
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e148c8cf/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
--
diff --git
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index 113dfd6..d14a852 100644
---
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -2442,7 +2442,12 @@ public class NotebookServer extends WebSocketServlet
Message resp = new Message(OP.EDITOR_SETTING);
resp.put("paragraphId", paragraphId);
Interpreter interpreter =
-notebook().getInterpreterFactory().getInterpreter(user, noteId,
replName);
+null;
+try {
+ interpreter = notebook().getInterpreterFactory().getInterpreter(user,
noteId, replName);
+} catch (InterpreterNotFoundException e) {
+ throw new IOException("Fail to get interpreter: " + replName, e);
+}
resp.put("editor", notebook().getInterpreterSettingManager().
getEditorSetting(interpreter, user, noteId, replName));
conn.send(serializeMessage(resp));
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e148c8cf/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
index 0af15ab..399aea8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Hel
zeppelin git commit: ZEPPELIN-3291. Throw exception instead of return null for InterpreterFactory.getInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 226e5b8b6 -> f49bd8c1c
ZEPPELIN-3291. Throw exception instead of return null for
InterpreterFactory.getInterpreter
### What is this PR for?
Minor code refactoring to throw exception when no interpreter is found
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3291
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2836 from zjffdu/ZEPPELIN-3291 and squashes the following commits:
bee319c [Jeff Zhang] ZEPPELIN-3291. Throw exception instead of return null for
InterpreterFactory.getInterpreter
(cherry picked from commit e148c8cfff04a7a299b2d626759a662a3f91087b)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/f49bd8c1
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/f49bd8c1
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/f49bd8c1
Branch: refs/heads/branch-0.8
Commit: f49bd8c1c18cc2386aa012ee83bd71d7c3a13629
Parents: 226e5b8
Author: Jeff Zhang
Authored: Mon Mar 5 12:58:19 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 10:08:44 2018 +0800
--
.../zeppelin/helium/ApplicationException.java | 4 ++
.../apache/zeppelin/socket/NotebookServer.java | 7 ++-
.../java/org/apache/zeppelin/helium/Helium.java | 7 ++-
.../helium/HeliumApplicationFactory.java| 24 ++---
.../interpreter/InterpreterFactory.java | 20 +++
.../InterpreterNotFoundException.java | 22
.../org/apache/zeppelin/notebook/Paragraph.java | 56 +++-
.../helium/HeliumApplicationFactoryTest.java| 11 +++-
.../interpreter/InterpreterFactoryTest.java | 12 +++--
.../InterpreterSettingManagerTest.java | 8 +--
.../org/apache/zeppelin/notebook/NoteTest.java | 11 ++--
.../apache/zeppelin/notebook/NotebookTest.java | 12 +++--
12 files changed, 133 insertions(+), 61 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/f49bd8c1/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
index 4bf0ac2..d3c6488 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/helium/ApplicationException.java
@@ -31,4 +31,8 @@ public class ApplicationException extends Exception {
public ApplicationException() {
}
+
+ public ApplicationException(String message, Throwable cause) {
+super(message, cause);
+ }
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/f49bd8c1/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
--
diff --git
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index 113dfd6..d14a852 100644
---
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -2442,7 +2442,12 @@ public class NotebookServer extends WebSocketServlet
Message resp = new Message(OP.EDITOR_SETTING);
resp.put("paragraphId", paragraphId);
Interpreter interpreter =
-notebook().getInterpreterFactory().getInterpreter(user, noteId,
replName);
+null;
+try {
+ interpreter = notebook().getInterpreterFactory().getInterpreter(user,
noteId, replName);
+} catch (InterpreterNotFoundException e) {
+ throw new IOException("Fail to get interpreter: " + replName, e);
+}
resp.put("editor", notebook().getInterpreterSettingManager().
getEditorSetting(interpreter, user, noteId, replName));
conn.send(serializeMessage(resp));
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/f49bd8c1/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/helium/Helium.java
index 0af15ab..399aea8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/
zeppelin git commit: [ZEPPELIN-3243] fix IndexOutOfBoundsException when 'No interpreter is binded to this note'
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 f49bd8c1c -> ec919123a
[ZEPPELIN-3243] fix IndexOutOfBoundsException when 'No interpreter is binded to
this note'
### What is this PR for?
I observed IndexOutOfBoundsException when using zeppelin locally on an existing
notebook repository and with fresh interpreter.json file.
The default Zeppelin behavior when using existing notebooks is the manual
binding of the interpreter settings, which works fine. This PR simply avoids
the IndexOutOfBoundsException.
### What type of PR is it?
Bug Fix
### Todos
-
### What is the Jira issue?
[https://issues.apache.org/jira/browse/ZEPPELIN-3243](https://issues.apache.org/jira/browse/ZEPPELIN-3243)
### How should this be tested?
- manual
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Andreas Weise
Closes #2808 from weand/ZEPPELIN-3243 and squashes the following commits:
b97a38e [Andreas Weise] ZEPPELIN-3243 fix IndexOutOfBoundsException when 'No
interpreter is binded to this note'
(cherry picked from commit fe660abb9663173998d6e02504ef3238f869c080)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/ec919123
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/ec919123
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/ec919123
Branch: refs/heads/branch-0.8
Commit: ec919123a888c78ca725fe86554e5313662a3c2a
Parents: f49bd8c
Author: Andreas Weise
Authored: Sun Feb 18 12:55:32 2018 +0100
Committer: Jeff Zhang
Committed: Tue Mar 6 10:13:19 2018 +0800
--
.../apache/zeppelin/interpreter/InterpreterSettingManager.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/ec919123/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 711812e..c3ec81b 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -412,7 +412,8 @@ public class InterpreterSettingManager {
@VisibleForTesting
public InterpreterSetting getDefaultInterpreterSetting(String noteId) {
-return getInterpreterSettings(noteId).get(0);
+List allInterpreterSettings =
getInterpreterSettings(noteId);
+return allInterpreterSettings.size() > 0 ? allInterpreterSettings.get(0) :
null;
}
public List getInterpreterSettings(String noteId) {
@@ -468,7 +469,7 @@ public class InterpreterSettingManager {
group = replNameSplit[0];
}
// when replName is 'name' of interpreter
-if (defaultSettingName.equals(intpSetting.getName())) {
+if (intpSetting.getName().equals(defaultSettingName)) {
editor =
intpSetting.getEditorFromSettingByClassName(interpreter.getClassName());
}
// when replName is 'alias name' of interpreter or 'group' of
interpreter
zeppelin git commit: [ZEPPELIN-3243] fix IndexOutOfBoundsException when 'No interpreter is binded to this note'
Repository: zeppelin
Updated Branches:
refs/heads/master e148c8cff -> fe660abb9
[ZEPPELIN-3243] fix IndexOutOfBoundsException when 'No interpreter is binded to
this note'
### What is this PR for?
I observed IndexOutOfBoundsException when using zeppelin locally on an existing
notebook repository and with fresh interpreter.json file.
The default Zeppelin behavior when using existing notebooks is the manual
binding of the interpreter settings, which works fine. This PR simply avoids
the IndexOutOfBoundsException.
### What type of PR is it?
Bug Fix
### Todos
-
### What is the Jira issue?
[https://issues.apache.org/jira/browse/ZEPPELIN-3243](https://issues.apache.org/jira/browse/ZEPPELIN-3243)
### How should this be tested?
- manual
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Andreas Weise
Closes #2808 from weand/ZEPPELIN-3243 and squashes the following commits:
b97a38e [Andreas Weise] ZEPPELIN-3243 fix IndexOutOfBoundsException when 'No
interpreter is binded to this note'
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/fe660abb
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/fe660abb
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/fe660abb
Branch: refs/heads/master
Commit: fe660abb9663173998d6e02504ef3238f869c080
Parents: e148c8c
Author: Andreas Weise
Authored: Sun Feb 18 12:55:32 2018 +0100
Committer: Jeff Zhang
Committed: Tue Mar 6 10:11:35 2018 +0800
--
.../apache/zeppelin/interpreter/InterpreterSettingManager.java | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/fe660abb/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 711812e..c3ec81b 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -412,7 +412,8 @@ public class InterpreterSettingManager {
@VisibleForTesting
public InterpreterSetting getDefaultInterpreterSetting(String noteId) {
-return getInterpreterSettings(noteId).get(0);
+List allInterpreterSettings =
getInterpreterSettings(noteId);
+return allInterpreterSettings.size() > 0 ? allInterpreterSettings.get(0) :
null;
}
public List getInterpreterSettings(String noteId) {
@@ -468,7 +469,7 @@ public class InterpreterSettingManager {
group = replNameSplit[0];
}
// when replName is 'name' of interpreter
-if (defaultSettingName.equals(intpSetting.getName())) {
+if (intpSetting.getName().equals(defaultSettingName)) {
editor =
intpSetting.getEditorFromSettingByClassName(interpreter.getClassName());
}
// when replName is 'alias name' of interpreter or 'group' of
interpreter
zeppelin git commit: ZEPPELIN-3279. [FlakyTest] NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
Repository: zeppelin
Updated Branches:
refs/heads/master fe660abb9 -> a6cd0e24c
ZEPPELIN-3279. [FlakyTest]
NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
### What is this PR for?
The root cause of this flaky test is that hashCode doesn't represent an unique
id of object. This PR just use IdGenerator for unique id and also combine
process id.
This PR also fix the flaky test caused by matplotlib.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3279
### How should this be tested?
* CI Pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2837 from zjffdu/ZEPPELIN-3279 and squashes the following commits:
063d4ee [Jeff Zhang] ZEPPELIN-3279. [FlakyTest]
NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a6cd0e24
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a6cd0e24
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a6cd0e24
Branch: refs/heads/master
Commit: a6cd0e24cc795985e200ebebf04de981e13de97c
Parents: fe660ab
Author: Jeff Zhang
Authored: Mon Mar 5 14:38:00 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 12:48:46 2018 +0800
--
testing/install_external_dependencies.sh | 2 +-
.../zeppelin/interpreter/mock/MockInterpreter1.java | 11 ++-
.../java/org/apache/zeppelin/notebook/NotebookTest.java | 5 +
3 files changed, 16 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a6cd0e24/testing/install_external_dependencies.sh
--
diff --git a/testing/install_external_dependencies.sh
b/testing/install_external_dependencies.sh
index d6c0736..a120d61 100755
--- a/testing/install_external_dependencies.sh
+++ b/testing/install_external_dependencies.sh
@@ -44,6 +44,6 @@ if [[ -n "$PYTHON" ]] ; then
conda update -q conda
conda info -a
conda config --add channels conda-forge
- conda install -q matplotlib pandasql ipython=5.4.1 jupyter_client ipykernel
matplotlib bokeh=0.12.10
+ conda install -q matplotlib=2.1.2 pandasql ipython=5.4.1 jupyter_client
ipykernel matplotlib bokeh=0.12.10
pip install -q grpcio ggplot bkzep==0.4.0
fi
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a6cd0e24/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
index a533c12..500c4f7 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
@@ -24,18 +24,27 @@ import
org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
+import java.lang.management.ManagementFactory;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
public class MockInterpreter1 extends Interpreter {
+ private static AtomicInteger IdGenerator = new AtomicInteger();
+
+ private int object_id;
+ private String pid;
Map vars = new HashMap<>();
public MockInterpreter1(Properties property) {
super(property);
+ this.object_id = IdGenerator.getAndIncrement();
+ this.pid =
ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
}
+
boolean open;
@@ -60,7 +69,7 @@ public class MockInterpreter1 extends Interpreter {
if ("getId".equals(st)) {
// get unique id of this interpreter instance
- result = new
InterpreterResult(InterpreterResult.Code.SUCCESS, "" + this.hashCode());
+ result = new
InterpreterResult(InterpreterResult.Code.SUCCESS, "" + this.object_id + "-" +
this.pid);
} else if (st.startsWith("sleep")) {
try {
Thread.sleep(Integer.parseInt(st.split("
")[1]));
http://git-wip-us.apache.org/repos/asf/zeppel
zeppelin git commit: ZEPPELIN-3279. [FlakyTest] NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 ec919123a -> 574de381a
ZEPPELIN-3279. [FlakyTest]
NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
### What is this PR for?
The root cause of this flaky test is that hashCode doesn't represent an unique
id of object. This PR just use IdGenerator for unique id and also combine
process id.
This PR also fix the flaky test caused by matplotlib.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3279
### How should this be tested?
* CI Pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2837 from zjffdu/ZEPPELIN-3279 and squashes the following commits:
063d4ee [Jeff Zhang] ZEPPELIN-3279. [FlakyTest]
NotebookTest.testPerSessionInterpreterCloseOnUnbindInterpreterSetting
(cherry picked from commit a6cd0e24cc795985e200ebebf04de981e13de97c)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/574de381
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/574de381
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/574de381
Branch: refs/heads/branch-0.8
Commit: 574de381a89d76ea0c5e80dc9ff3ca3debe7a01d
Parents: ec91912
Author: Jeff Zhang
Authored: Mon Mar 5 14:38:00 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 12:49:03 2018 +0800
--
testing/install_external_dependencies.sh | 2 +-
.../zeppelin/interpreter/mock/MockInterpreter1.java | 11 ++-
.../java/org/apache/zeppelin/notebook/NotebookTest.java | 5 +
3 files changed, 16 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/574de381/testing/install_external_dependencies.sh
--
diff --git a/testing/install_external_dependencies.sh
b/testing/install_external_dependencies.sh
index d6c0736..a120d61 100755
--- a/testing/install_external_dependencies.sh
+++ b/testing/install_external_dependencies.sh
@@ -44,6 +44,6 @@ if [[ -n "$PYTHON" ]] ; then
conda update -q conda
conda info -a
conda config --add channels conda-forge
- conda install -q matplotlib pandasql ipython=5.4.1 jupyter_client ipykernel
matplotlib bokeh=0.12.10
+ conda install -q matplotlib=2.1.2 pandasql ipython=5.4.1 jupyter_client
ipykernel matplotlib bokeh=0.12.10
pip install -q grpcio ggplot bkzep==0.4.0
fi
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/574de381/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
index a533c12..500c4f7 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/mock/MockInterpreter1.java
@@ -24,18 +24,27 @@ import
org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
+import java.lang.management.ManagementFactory;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
+import java.util.concurrent.atomic.AtomicInteger;
public class MockInterpreter1 extends Interpreter {
+ private static AtomicInteger IdGenerator = new AtomicInteger();
+
+ private int object_id;
+ private String pid;
Map vars = new HashMap<>();
public MockInterpreter1(Properties property) {
super(property);
+ this.object_id = IdGenerator.getAndIncrement();
+ this.pid =
ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
}
+
boolean open;
@@ -60,7 +69,7 @@ public class MockInterpreter1 extends Interpreter {
if ("getId".equals(st)) {
// get unique id of this interpreter instance
- result = new
InterpreterResult(InterpreterResult.Code.SUCCESS, "" + this.hashCode());
+ result = new
InterpreterResult(InterpreterResult.Code.SUCCESS, "" + this.object_id + "-" +
this.pid);
} else if (st.startsWith("sleep")) {
try {
Thread.sleep(In
zeppelin git commit: ZEPPELIN-3277. NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent fails
Repository: zeppelin
Updated Branches:
refs/heads/master a6cd0e24c -> c77c549e3
ZEPPELIN-3277.
NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent
fails
### What is this PR for?
Trivial fix for flaky test, see the comment for details.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3277
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2838 from zjffdu/ZEPPELIN-3277 and squashes the following commits:
68827d7 [Jeff Zhang] ZEPPELIN-3277.
NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent
fails
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c77c549e
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c77c549e
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c77c549e
Branch: refs/heads/master
Commit: c77c549e3f6bfa7b08ebc0bfaacb2bd8719c875f
Parents: a6cd0e2
Author: Jeff Zhang
Authored: Mon Mar 5 15:52:14 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 12:52:33 2018 +0800
--
.../test/java/org/apache/zeppelin/socket/NotebookServerTest.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c77c549e/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
--
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
index a0cb587..0b68ca6 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
@@ -123,6 +123,8 @@ public class NotebookServerTest extends AbstractTestRestApi
{
}
Thread.sleep(100);
}
+// sleep for 1 second to make sure job running thread finish to fire
event. See ZEPPELIN-3277
+Thread.sleep(1000);
// add angularObject
interpreterGroup.getAngularObjectRegistry().add("object1", "value1",
note1.getId(), null);
zeppelin git commit: ZEPPELIN-3277. NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent fails
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 574de381a -> 3b3ce9ed8
ZEPPELIN-3277.
NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent
fails
### What is this PR for?
Trivial fix for flaky test, see the comment for details.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3277
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2838 from zjffdu/ZEPPELIN-3277 and squashes the following commits:
68827d7 [Jeff Zhang] ZEPPELIN-3277.
NotebookServerTest.testMakeSureNoAngularObjectBroadcastToWebsocketWhoFireTheEvent
fails
(cherry picked from commit c77c549e3f6bfa7b08ebc0bfaacb2bd8719c875f)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/3b3ce9ed
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/3b3ce9ed
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/3b3ce9ed
Branch: refs/heads/branch-0.8
Commit: 3b3ce9ed817273807959a97340598b8dc06d8f96
Parents: 574de38
Author: Jeff Zhang
Authored: Mon Mar 5 15:52:14 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 6 12:52:49 2018 +0800
--
.../test/java/org/apache/zeppelin/socket/NotebookServerTest.java | 2 ++
1 file changed, 2 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3b3ce9ed/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
--
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
index a0cb587..0b68ca6 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/socket/NotebookServerTest.java
@@ -123,6 +123,8 @@ public class NotebookServerTest extends AbstractTestRestApi
{
}
Thread.sleep(100);
}
+// sleep for 1 second to make sure job running thread finish to fire
event. See ZEPPELIN-3277
+Thread.sleep(1000);
// add angularObject
interpreterGroup.getAngularObjectRegistry().add("object1", "value1",
note1.getId(), null);
zeppelin git commit: ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11
Repository: zeppelin Updated Branches: refs/heads/master c77c549e3 -> 0cff6f0f0 ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11 ### What is this PR for? Trivial PR to remove the build of profile spark-1.6 and scala-2.11, The travis build of profile spark-1.6 and scala-2.11 only test the embedded mode of spark instead real spark binary distribution of spark 1.6 with scala-2.11, so IMHO it is safe for us to just remove that build and only keep spark 1.6 with scala-2.10 ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3295 ### How should this be tested? * Travis CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2840 from zjffdu/ZEPPELIN-3295 and squashes the following commits: 946cf5d [Jeff Zhang] ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11 Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/0cff6f0f Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/0cff6f0f Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/0cff6f0f Branch: refs/heads/master Commit: 0cff6f0f06570fdf331c102377402ac9732282e5 Parents: c77c549 Author: Jeff Zhang Authored: Tue Mar 6 22:29:41 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 07:36:45 2018 +0800 -- .travis.yml | 5 - 1 file changed, 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0cff6f0f/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index aba2944..a6f72c8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -114,11 +114,6 @@ matrix: dist: trusty env: PYTHON="3" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test spark module for 1.6.3 with scala 2.11 -- jdk: "oraclejdk8" - dist: trusty - env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" - # Test python/pyspark with python 2, livy 0.5 - sudo: required dist: trusty
zeppelin git commit: ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 8d06dee3b -> 95f600a5a ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11 ### What is this PR for? Trivial PR to remove the build of profile spark-1.6 and scala-2.11, The travis build of profile spark-1.6 and scala-2.11 only test the embedded mode of spark instead real spark binary distribution of spark 1.6 with scala-2.11, so IMHO it is safe for us to just remove that build and only keep spark 1.6 with scala-2.10 ### What type of PR is it? [Improvement] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3295 ### How should this be tested? * Travis CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2840 from zjffdu/ZEPPELIN-3295 and squashes the following commits: 946cf5d [Jeff Zhang] ZEPPELIN-3295. Remove build of profile spark-1.6 and scala-2.11 (cherry picked from commit 0cff6f0f06570fdf331c102377402ac9732282e5) Signed-off-by: Jeff Zhang Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/95f600a5 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/95f600a5 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/95f600a5 Branch: refs/heads/branch-0.8 Commit: 95f600a5a7888cb0a6f8d2c958f904519abf209e Parents: 8d06dee Author: Jeff Zhang Authored: Tue Mar 6 22:29:41 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 07:36:57 2018 +0800 -- .travis.yml | 5 - 1 file changed, 5 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/95f600a5/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index df96c38..662386f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -114,11 +114,6 @@ matrix: dist: trusty env: PYTHON="3" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test spark module for 1.6.3 with scala 2.11 -- jdk: "oraclejdk8" - dist: trusty - env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" - # Test python/pyspark with python 2, livy 0.5 - sudo: required dist: trusty
zeppelin git commit: ZEPPELIN-3296. Reorg livy integration test to minimize livy session
Repository: zeppelin Updated Branches: refs/heads/master 0cff6f0f0 -> 483dc3f2b ZEPPELIN-3296. Reorg livy integration test to minimize livy session ### What is this PR for? Just refactor livy integration test to minuze livy session so that we can reduce the livy build time. ### What type of PR is it? [Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3296 ### How should this be tested? * Travis CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2844 from zjffdu/ZEPPELIN-3296 and squashes the following commits: 206ea3e [Jeff Zhang] ZEPPELIN-3296. Reorg livy integration test to minimize livy session Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/483dc3f2 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/483dc3f2 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/483dc3f2 Branch: refs/heads/master Commit: 483dc3f2bb46d18b7bbb41d72118c356bd9de403 Parents: 0cff6f0 Author: Jeff Zhang Authored: Tue Mar 6 17:17:35 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 09:24:40 2018 +0800 -- .travis.yml | 9 +- .../apache/zeppelin/livy/LivyInterpreterIT.java | 646 +++ 2 files changed, 250 insertions(+), 405 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/483dc3f2/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index a6f72c8..9edb198 100644 --- a/.travis.yml +++ b/.travis.yml @@ -114,17 +114,16 @@ matrix: dist: trusty env: PYTHON="3" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test python/pyspark with python 2, livy 0.5 - sudo: required dist: trusty jdk: "openjdk7" - env: PYTHON="2" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="-Dpyspark.test.exclude='' -DfailIfNoTests=false" + env: PYTHON="2" SPARK_VER="1.6.3" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="" -# Test python/pyspark with python 3, livy 0.5 +# Test livy 0.5 with spark 2.2.0 under python3 - sudo: required dist: trusty - jdk: "openjdk7" - env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.0.0" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="-Pspark-2.0 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="-Dpyspark.test.exclude='' -DfailIfNoTests=false" + jdk: "openjdk8" + env: PYTHON="3" SPARK_VER="2.2.0" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="" before_install: # check files included in commit range, clear bower_components if a bower.json file has changed. http://git-wip-us.apache.org/repos/asf/zeppelin/blob/483dc3f2/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java -- diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java index 3dfeb36..96fdbea 100644 --- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java +++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java @@ -18,17 +18,27 @@ package org.apache.zeppelin.livy; +import org.apache.commons.io.IOUtils; import org.
zeppelin git commit: ZEPPELIN-3296. Reorg livy integration test to minimize livy session
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 95f600a5a -> c5a49a471 ZEPPELIN-3296. Reorg livy integration test to minimize livy session ### What is this PR for? Just refactor livy integration test to minuze livy session so that we can reduce the livy build time. ### What type of PR is it? [Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3296 ### How should this be tested? * Travis CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2844 from zjffdu/ZEPPELIN-3296 and squashes the following commits: 206ea3e [Jeff Zhang] ZEPPELIN-3296. Reorg livy integration test to minimize livy session (cherry picked from commit 483dc3f2bb46d18b7bbb41d72118c356bd9de403) Signed-off-by: Jeff Zhang Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c5a49a47 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c5a49a47 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c5a49a47 Branch: refs/heads/branch-0.8 Commit: c5a49a471eb9cf4f8ebe56a0eac2ee1ae3c7c732 Parents: 95f600a Author: Jeff Zhang Authored: Tue Mar 6 17:17:35 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 09:24:53 2018 +0800 -- .travis.yml | 9 +- .../apache/zeppelin/livy/LivyInterpreterIT.java | 646 +++ 2 files changed, 250 insertions(+), 405 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c5a49a47/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index 662386f..63097b8 100644 --- a/.travis.yml +++ b/.travis.yml @@ -114,17 +114,16 @@ matrix: dist: trusty env: PYTHON="3" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test python/pyspark with python 2, livy 0.5 - sudo: required dist: trusty jdk: "openjdk7" - env: PYTHON="2" SCALA_VER="2.10" SPARK_VER="1.6.3" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="-Pspark-1.6 -Phadoop2 -Phadoop-2.6 -Pscala-2.10" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="-Dpyspark.test.exclude='' -DfailIfNoTests=false" + env: PYTHON="2" SPARK_VER="1.6.3" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="" -# Test python/pyspark with python 3, livy 0.5 +# Test livy 0.5 with spark 2.2.0 under python3 - sudo: required dist: trusty - jdk: "openjdk7" - env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.0.0" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="-Pspark-2.0 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="-Dpyspark.test.exclude='' -DfailIfNoTests=false" + jdk: "openjdk8" + env: PYTHON="3" SPARK_VER="2.2.0" HADOOP_VER="2.6" LIVY_VER="0.5.0-incubating" PROFILE="" BUILD_FLAG="install -am -DskipTests -DskipRat" TEST_FLAG="verify -DskipRat" MODULES="-pl livy" TEST_PROJECTS="" before_install: # check files included in commit range, clear bower_components if a bower.json file has changed. http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c5a49a47/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java -- diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java index 3dfeb36..96fdbea 100644 --- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java +++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInt
zeppelin git commit: ZEPPELIN-3299. Combine spark integration test with its unit test in one build
Repository: zeppelin Updated Branches: refs/heads/master 483dc3f2b -> 733a20210 ZEPPELIN-3299. Combine spark integration test with its unit test in one build ### What is this PR for? In ZEPPELIN-3254, we introduce 2 build for spark integration test. And we can run unit test with these system test together so that we could save 2 build in travis and also save time to set up enviroment. ### What type of PR is it? [Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3299 ### How should this be tested? * Travis CI Pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2843 from zjffdu/ZEPPELIN-3299 and squashes the following commits: dcbbb10 [Jeff Zhang] ZEPPELIN-3299. Combine spark integration test with its unit test in one build Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/733a2021 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/733a2021 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/733a2021 Branch: refs/heads/master Commit: 733a202109d9087556f24822347a3903da997a88 Parents: 483dc3f Author: Jeff Zhang Authored: Tue Mar 6 22:33:30 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 11:15:23 2018 +0800 -- .travis.yml | 19 +-- 1 file changed, 5 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/733a2021/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index 9edb198..bcef80a 100644 --- a/.travis.yml +++ b/.travis.yml @@ -82,22 +82,17 @@ matrix: # Run ZeppelinSparkClusterTest & SparkIntegrationTest in one build would exceed the time limitation of travis, so running them separately -# Test spark interpreter with different spark versions under python2, only run ZeppelinSparkClusterTest +# Integration test of spark interpreter with different spark versions under python2, only run ZeppelinSparkClusterTest. Also run spark unit test of spark 2.2 in this build. - sudo: required jdk: "oraclejdk8" dist: trusty - env: PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,zeppelin-server,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest -DfailIfNoTests=false" + env: PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,zeppelin-server,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest,org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test spark interpreter with different spark versions under python3, only run SparkIntegrationTest +# Integration test of spark interpreter with different spark versions under python3, only run SparkIntegrationTest. Also run spark unit test of spark 1.6 in this build. - sudo: required jdk: "oraclejdk8" dist: trusty - env: PYTHON="3" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTest -DfailIfNoTests=false" - -# Test spark module for 2.2.0 with scala 2.11 -- jdk: "oraclejdk8" - dist: trusty - env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6" PROFILE="-Pspark-2.2 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" + env: PYTHON="3" PROFILE="-Pspark-1.6" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTest,org.apache.zeppelin.spark.* -DfailIfNoTests=false" # Test spark module for 2.1.0 with scala 2.11 - jdk:
zeppelin git commit: ZEPPELIN-3299. Combine spark integration test with its unit test in one build
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 c5a49a471 -> dc4612ed1 ZEPPELIN-3299. Combine spark integration test with its unit test in one build ### What is this PR for? In ZEPPELIN-3254, we introduce 2 build for spark integration test. And we can run unit test with these system test together so that we could save 2 build in travis and also save time to set up enviroment. ### What type of PR is it? [Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3299 ### How should this be tested? * Travis CI Pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2843 from zjffdu/ZEPPELIN-3299 and squashes the following commits: dcbbb10 [Jeff Zhang] ZEPPELIN-3299. Combine spark integration test with its unit test in one build (cherry picked from commit 733a202109d9087556f24822347a3903da997a88) Signed-off-by: Jeff Zhang Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/dc4612ed Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/dc4612ed Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/dc4612ed Branch: refs/heads/branch-0.8 Commit: dc4612ed175daa4fbf6386b6637544df60945ec8 Parents: c5a49a4 Author: Jeff Zhang Authored: Tue Mar 6 22:33:30 2018 +0800 Committer: Jeff Zhang Committed: Wed Mar 7 11:15:35 2018 +0800 -- .travis.yml | 19 +-- 1 file changed, 5 insertions(+), 14 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/dc4612ed/.travis.yml -- diff --git a/.travis.yml b/.travis.yml index 63097b8..732029c 100644 --- a/.travis.yml +++ b/.travis.yml @@ -82,22 +82,17 @@ matrix: # Run ZeppelinSparkClusterTest & SparkIntegrationTest in one build would exceed the time limitation of travis, so running them separately -# Test spark interpreter with different spark versions under python2, only run ZeppelinSparkClusterTest +# Integration test of spark interpreter with different spark versions under python2, only run ZeppelinSparkClusterTest. Also run spark unit test of spark 2.2 in this build. - sudo: required jdk: "oraclejdk8" dist: trusty - env: PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,zeppelin-server,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest -DfailIfNoTests=false" + env: PYTHON="2" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,zeppelin-server,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=ZeppelinSparkClusterTest,org.apache.zeppelin.spark.* -DfailIfNoTests=false" -# Test spark interpreter with different spark versions under python3, only run SparkIntegrationTest +# Integration test of spark interpreter with different spark versions under python3, only run SparkIntegrationTest. Also run spark unit test of spark 1.6 in this build. - sudo: required jdk: "oraclejdk8" dist: trusty - env: PYTHON="3" PROFILE="-Pspark-2.2" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTest -DfailIfNoTests=false" - -# Test spark module for 2.2.0 with scala 2.11 -- jdk: "oraclejdk8" - dist: trusty - env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6" PROFILE="-Pspark-2.2 -Phadoop3 -Phadoop-2.6 -Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false" + env: PYTHON="3" PROFILE="-Pspark-1.6" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am" MODULES="-pl zeppelin-zengine,spark/interpreter,spark/spark-dependencies" TEST_PROJECTS="-Dtest=SparkIntegrationTest,org.apache.zeppelin.spa
zeppelin git commit: ZEPPELIN-3302. Update SparkVersion.java to support Spark 2.3
Repository: zeppelin
Updated Branches:
refs/heads/master 733a20210 -> db3774640
ZEPPELIN-3302. Update SparkVersion.java to support Spark 2.3
### What is this PR for?
Trivial PR to support 2.3 via updating SparkVersion.java. Unfortunately we
cannot download spark 2.3 from http://d3kbcqa49mib13.cloudfront.net, will leave
it in ZEPPELIN-3305
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3302
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2846 from zjffdu/ZEPPELIN-3302-2 and squashes the following commits:
212eb68 [Jeff Zhang] ZEPPELIN-3302. Update SparkVersion.java to support Spark
2.3
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/db377464
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/db377464
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/db377464
Branch: refs/heads/master
Commit: db3774640f446db9c6b00ee8ca5375405dd027be
Parents: 733a202
Author: Jeff Zhang
Authored: Wed Mar 7 11:21:19 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 13:35:44 2018 +0800
--
.../src/main/java/org/apache/zeppelin/spark/SparkVersion.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/db377464/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
index 4b02798..09ea332 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
@@ -34,10 +34,10 @@ public class SparkVersion {
public static final SparkVersion SPARK_1_6_0 =
SparkVersion.fromVersionString("1.6.0");
public static final SparkVersion SPARK_2_0_0 =
SparkVersion.fromVersionString("2.0.0");
- public static final SparkVersion SPARK_2_3_0 =
SparkVersion.fromVersionString("2.3.0");
+ public static final SparkVersion SPARK_2_4_0 =
SparkVersion.fromVersionString("2.4.0");
public static final SparkVersion MIN_SUPPORTED_VERSION = SPARK_1_0_0;
- public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_2_3_0;
+ public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_2_4_0;
private int version;
private String versionString;
zeppelin git commit: ZEPPELIN-3302. Update SparkVersion.java to support Spark 2.3
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 dc4612ed1 -> 891108ba0
ZEPPELIN-3302. Update SparkVersion.java to support Spark 2.3
### What is this PR for?
Trivial PR to support 2.3 via updating SparkVersion.java. Unfortunately we
cannot download spark 2.3 from http://d3kbcqa49mib13.cloudfront.net, will leave
it in ZEPPELIN-3305
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3302
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2846 from zjffdu/ZEPPELIN-3302-2 and squashes the following commits:
212eb68 [Jeff Zhang] ZEPPELIN-3302. Update SparkVersion.java to support Spark
2.3
(cherry picked from commit db3774640f446db9c6b00ee8ca5375405dd027be)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/891108ba
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/891108ba
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/891108ba
Branch: refs/heads/branch-0.8
Commit: 891108ba0d236b955a1d7c552dae5577ea0bacec
Parents: dc4612e
Author: Jeff Zhang
Authored: Wed Mar 7 11:21:19 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 13:36:00 2018 +0800
--
.../src/main/java/org/apache/zeppelin/spark/SparkVersion.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/891108ba/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
index 4b02798..09ea332 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkVersion.java
@@ -34,10 +34,10 @@ public class SparkVersion {
public static final SparkVersion SPARK_1_6_0 =
SparkVersion.fromVersionString("1.6.0");
public static final SparkVersion SPARK_2_0_0 =
SparkVersion.fromVersionString("2.0.0");
- public static final SparkVersion SPARK_2_3_0 =
SparkVersion.fromVersionString("2.3.0");
+ public static final SparkVersion SPARK_2_4_0 =
SparkVersion.fromVersionString("2.4.0");
public static final SparkVersion MIN_SUPPORTED_VERSION = SPARK_1_0_0;
- public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_2_3_0;
+ public static final SparkVersion UNSUPPORTED_FUTURE_VERSION = SPARK_2_4_0;
private int version;
private String versionString;
zeppelin git commit: ZEPPELIN-3304. Move sap module before zeppelin-distribution
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 891108ba0 -> 591483c6a
ZEPPELIN-3304. Move sap module before zeppelin-distribution
### What is this PR for?
1. Move sap module before zeppelin-distribution, otherwise sap module will be
built after zeppelin-distribution
2. Fix code style issue of sap module
3. Update travis to include sap module
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3304
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2845 from zjffdu/ZEPPELIN-3304 and squashes the following commits:
533a1b0 [Jeff Zhang] ZEPPELIN-3304. Move sap module before zeppelin-distribution
(cherry picked from commit a3952ea95c72543ca767d2c9dd2f4c6cab63fe1b)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/591483c6
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/591483c6
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/591483c6
Branch: refs/heads/branch-0.8
Commit: 591483c6a679836a727a500592a3f93e9c45bfc1
Parents: 891108b
Author: Jeff Zhang
Authored: Wed Mar 7 11:54:51 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 13:45:20 2018 +0800
--
.travis.yml | 2 +-
pom.xml | 2 +-
.../main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java | 3 ++-
3 files changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/591483c6/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index 732029c..a1c2e52 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -37,7 +37,7 @@ addons:
env:
global:
# Interpreters does not required by zeppelin-server integration tests
--
INTERPRETERS='!hbase,!pig,!jdbc,!file,!flink,!ignite,!kylin,!lens,!cassandra,!elasticsearch,!bigquery,!alluxio,!scio,!livy,!groovy'
+-
INTERPRETERS='!hbase,!pig,!jdbc,!file,!flink,!ignite,!kylin,!lens,!cassandra,!elasticsearch,!bigquery,!alluxio,!scio,!livy,!groovy,!sap'
matrix:
include:
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/591483c6/pom.xml
--
diff --git a/pom.xml b/pom.xml
index aad313d..efb1abd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77,11 +77,11 @@
alluxio
scio
neo4j
+sap
zeppelin-web
zeppelin-server
zeppelin-jupyter
zeppelin-distribution
-sap
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/591483c6/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
--
diff --git
a/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
b/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
index dc9099d..31fbf51 100644
--- a/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
+++ b/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
@@ -188,7 +188,8 @@ public class UniverseUtil {
}
if (buf.toString().toLowerCase().endsWith("where") || i ==
array.length - 1) {
selectPart = false;
-
select.append(parseResultObj(resultObj.toString().replaceAll("(?i)wher$", ""),
nodeInfos));
+
select.append(parseResultObj(resultObj.toString().replaceAll("(?i)wher$", ""),
+ nodeInfos));
select.append(RESULT_END_TEMPLATE);
continue;
}
zeppelin git commit: ZEPPELIN-3304. Move sap module before zeppelin-distribution
Repository: zeppelin
Updated Branches:
refs/heads/master db3774640 -> a3952ea95
ZEPPELIN-3304. Move sap module before zeppelin-distribution
### What is this PR for?
1. Move sap module before zeppelin-distribution, otherwise sap module will be
built after zeppelin-distribution
2. Fix code style issue of sap module
3. Update travis to include sap module
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3304
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2845 from zjffdu/ZEPPELIN-3304 and squashes the following commits:
533a1b0 [Jeff Zhang] ZEPPELIN-3304. Move sap module before zeppelin-distribution
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a3952ea9
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a3952ea9
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a3952ea9
Branch: refs/heads/master
Commit: a3952ea95c72543ca767d2c9dd2f4c6cab63fe1b
Parents: db37746
Author: Jeff Zhang
Authored: Wed Mar 7 11:54:51 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 13:45:07 2018 +0800
--
.travis.yml | 2 +-
pom.xml | 2 +-
.../main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java | 3 ++-
3 files changed, 4 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a3952ea9/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index bcef80a..dcd6cea 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -37,7 +37,7 @@ addons:
env:
global:
# Interpreters does not required by zeppelin-server integration tests
--
INTERPRETERS='!hbase,!pig,!jdbc,!file,!flink,!ignite,!kylin,!lens,!cassandra,!elasticsearch,!bigquery,!alluxio,!scio,!livy,!groovy'
+-
INTERPRETERS='!hbase,!pig,!jdbc,!file,!flink,!ignite,!kylin,!lens,!cassandra,!elasticsearch,!bigquery,!alluxio,!scio,!livy,!groovy,!sap'
matrix:
include:
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a3952ea9/pom.xml
--
diff --git a/pom.xml b/pom.xml
index dc6d64b..d6087a8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -77,11 +77,11 @@
alluxio
scio
neo4j
+sap
zeppelin-web
zeppelin-server
zeppelin-jupyter
zeppelin-distribution
-sap
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a3952ea9/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
--
diff --git
a/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
b/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
index dc9099d..31fbf51 100644
--- a/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
+++ b/sap/src/main/java/org/apache/zeppelin/sap/universe/UniverseUtil.java
@@ -188,7 +188,8 @@ public class UniverseUtil {
}
if (buf.toString().toLowerCase().endsWith("where") || i ==
array.length - 1) {
selectPart = false;
-
select.append(parseResultObj(resultObj.toString().replaceAll("(?i)wher$", ""),
nodeInfos));
+
select.append(parseResultObj(resultObj.toString().replaceAll("(?i)wher$", ""),
+ nodeInfos));
select.append(RESULT_END_TEMPLATE);
continue;
}
zeppelin git commit: ZEPPELIN-3303. Avoid download spark for travis build
Repository: zeppelin
Updated Branches:
refs/heads/master a3952ea95 -> 2afcb944e
ZEPPELIN-3303. Avoid download spark for travis build
### What is this PR for?
It is not necessary to download spark distribution for several travis build.
This PR would remove that to save time.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3303
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2847 from zjffdu/ZEPPELIN-3303 and squashes the following commits:
b5c4049 [Jeff Zhang] ZEPPELIN-3303. Avoid download spark for travis build
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/2afcb944
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/2afcb944
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/2afcb944
Branch: refs/heads/master
Commit: 2afcb944e2240b477387498aeacc2dabab8f2842
Parents: a3952ea
Author: Jeff Zhang
Authored: Wed Mar 7 13:50:45 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 16:00:17 2018 +0800
--
.travis.yml | 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/2afcb944/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index dcd6cea..24476fb 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -44,7 +44,7 @@ matrix:
# Test License compliance using RAT tool
- jdk: "openjdk7"
dist: trusty
- env: SCALA_VER="2.11" SPARK_VER="2.0.2" HADOOP_VER="2.6" PROFILE="-Prat"
BUILD_FLAG="clean" TEST_FLAG="org.apache.rat:apache-rat-plugin:check"
TEST_PROJECTS=""
+ env: SCALA_VER="2.11" PROFILE="-Prat" BUILD_FLAG="clean"
TEST_FLAG="org.apache.rat:apache-rat-plugin:check" TEST_PROJECTS=""
# Run e2e tests (in zeppelin-web)
# chrome dropped the support for precise (ubuntu 12.04), so need to use
trusty
@@ -66,7 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding
-Phelium-dev -Pexamples -Pscala-2.11" BUILD_FLAG="package -Pbuild-distr
-DskipRat" TEST_FLAG="verify -Pusing-packaged-distr -DskipRat" MODULES="-pl
${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
@@ -97,12 +97,12 @@ matrix:
# Test spark module for 2.1.0 with scala 2.11
- jdk: "openjdk7"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.1.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.1 -Phadoop2 -Phadoop-2.6 -Pscala-2.11" SPARKR="true"
BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am"
MODULES="-pl spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false"
+ env: PYTHON="2" SCALA_VER="2.11" PROFILE="-Pspark-2.1 -Phadoop2
-Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am"
TEST_FLAG="test -DskipRat -am" MODULES="-pl
spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false"
# Test spark module for 2.0.2 with scala 2.11
- jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.0.2" HADOOP_VER="2.6"
PROFILE="-Pspark-2.0 -Phadoop3 -Phadoop-2.6 -Pscala-2.11&qu
zeppelin git commit: ZEPPELIN-3303. Avoid download spark for travis build
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 591483c6a -> a12aa7dee
ZEPPELIN-3303. Avoid download spark for travis build
### What is this PR for?
It is not necessary to download spark distribution for several travis build.
This PR would remove that to save time.
### What type of PR is it?
[ Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3303
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2847 from zjffdu/ZEPPELIN-3303 and squashes the following commits:
b5c4049 [Jeff Zhang] ZEPPELIN-3303. Avoid download spark for travis build
(cherry picked from commit 2afcb944e2240b477387498aeacc2dabab8f2842)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a12aa7de
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a12aa7de
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a12aa7de
Branch: refs/heads/branch-0.8
Commit: a12aa7dee88bcb453b1fb2b6b3204e18cb1fe0b2
Parents: 591483c
Author: Jeff Zhang
Authored: Wed Mar 7 13:50:45 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 16:00:32 2018 +0800
--
.travis.yml | 8
1 file changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a12aa7de/.travis.yml
--
diff --git a/.travis.yml b/.travis.yml
index a1c2e52..d2e3216 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -44,7 +44,7 @@ matrix:
# Test License compliance using RAT tool
- jdk: "openjdk7"
dist: trusty
- env: SCALA_VER="2.11" SPARK_VER="2.0.2" HADOOP_VER="2.6" PROFILE="-Prat"
BUILD_FLAG="clean" TEST_FLAG="org.apache.rat:apache-rat-plugin:check"
TEST_PROJECTS=""
+ env: SCALA_VER="2.11" PROFILE="-Prat" BUILD_FLAG="clean"
TEST_FLAG="org.apache.rat:apache-rat-plugin:check" TEST_PROJECTS=""
# Run e2e tests (in zeppelin-web)
# chrome dropped the support for precise (ubuntu 12.04), so need to use
trusty
@@ -66,7 +66,7 @@ matrix:
- sudo: required
jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="3" SCALA_VER="2.11" SPARK_VER="2.2.0" HADOOP_VER="2.6"
SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding -Phelium-dev -Pexamples
-Pscala-2.11" BUILD_FLAG="package -Pbuild-distr -DskipRat" TEST_FLAG="verify
-Pusing-packaged-distr -DskipRat" MODULES="-pl ${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
+ env: PYTHON="3" SPARKR="true" PROFILE="-Pspark-2.2 -Pscalding
-Phelium-dev -Pexamples -Pscala-2.11" BUILD_FLAG="package -Pbuild-distr
-DskipRat" TEST_FLAG="verify -Pusing-packaged-distr -DskipRat" MODULES="-pl
${INTERPRETERS}"
TEST_PROJECTS="-Dtests.to.exclude=**/SparkIntegrationTest.java,**/ZeppelinSparkClusterTest.java,**/org/apache/zeppelin/spark/*,**/HeliumApplicationFactoryTest.java
-DfailIfNoTests=false"
# Test selenium with spark module for 1.6.3
- jdk: "oraclejdk8"
@@ -97,12 +97,12 @@ matrix:
# Test spark module for 2.1.0 with scala 2.11
- jdk: "openjdk7"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER="2.1.0" HADOOP_VER="2.6"
PROFILE="-Pspark-2.1 -Phadoop2 -Phadoop-2.6 -Pscala-2.11" SPARKR="true"
BUILD_FLAG="package -DskipTests -DskipRat -am" TEST_FLAG="test -DskipRat -am"
MODULES="-pl spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false"
+ env: PYTHON="2" SCALA_VER="2.11" PROFILE="-Pspark-2.1 -Phadoop2
-Pscala-2.11" SPARKR="true" BUILD_FLAG="package -DskipTests -DskipRat -am"
TEST_FLAG="test -DskipRat -am" MODULES="-pl
spark/interpreter,spark/spark-dependencies"
TEST_PROJECTS="-Dtest=org.apache.zeppelin.spark.* -DfailIfNoTests=false"
# Test spark module for 2.0.2 with scala 2.11
- jdk: "oraclejdk8"
dist: trusty
- env: PYTHON="2" SCALA_VER="2.11" SPARK_VER=&
zeppelin git commit: ZEPPELIN-3286. Run All Paragraphs stops if there is a disabled paragraph mid-run
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 a12aa7dee -> 7a41c876b
ZEPPELIN-3286. Run All Paragraphs stops if there is a disabled paragraph mid-run
### What is this PR for?
Trivial fix that just skip the disable paragraph when run all paragraphs.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3286
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2833 from zjffdu/ZEPPELIN-3286 and squashes the following commits:
924d1fc [Jeff Zhang] ZEPPELIN-3286. Run All Paragraphs stops if there is a
disabled paragraph mid-run
(cherry picked from commit c904e56a8cfd2f848ccf9612c99edf4a1037a128)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7a41c876
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7a41c876
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7a41c876
Branch: refs/heads/branch-0.8
Commit: 7a41c876b0d0bb1105fca487f0c8436124d377b1
Parents: a12aa7d
Author: Jeff Zhang
Authored: Sun Mar 4 13:11:50 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 18:38:16 2018 +0800
--
.../src/main/java/org/apache/zeppelin/socket/NotebookServer.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a41c876/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
--
diff --git
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index d14a852..074 100644
---
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -1723,7 +1723,7 @@ public class NotebookServer extends WebSocketServlet
Paragraph p = setParagraphUsingMessage(note, fromMessage,
paragraphId, text, title, params, config);
- if (!persistAndExecuteSingleParagraph(conn, note, p, true)) {
+ if (p.isEnabled() && !persistAndExecuteSingleParagraph(conn, note, p,
true)) {
// stop execution when one paragraph fails.
break;
}
zeppelin git commit: ZEPPELIN-3286. Run All Paragraphs stops if there is a disabled paragraph mid-run
Repository: zeppelin
Updated Branches:
refs/heads/master 2afcb944e -> c904e56a8
ZEPPELIN-3286. Run All Paragraphs stops if there is a disabled paragraph mid-run
### What is this PR for?
Trivial fix that just skip the disable paragraph when run all paragraphs.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3286
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2833 from zjffdu/ZEPPELIN-3286 and squashes the following commits:
924d1fc [Jeff Zhang] ZEPPELIN-3286. Run All Paragraphs stops if there is a
disabled paragraph mid-run
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c904e56a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c904e56a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c904e56a
Branch: refs/heads/master
Commit: c904e56a8cfd2f848ccf9612c99edf4a1037a128
Parents: 2afcb94
Author: Jeff Zhang
Authored: Sun Mar 4 13:11:50 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 7 18:37:56 2018 +0800
--
.../src/main/java/org/apache/zeppelin/socket/NotebookServer.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c904e56a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
--
diff --git
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
index d14a852..074 100644
---
a/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
+++
b/zeppelin-server/src/main/java/org/apache/zeppelin/socket/NotebookServer.java
@@ -1723,7 +1723,7 @@ public class NotebookServer extends WebSocketServlet
Paragraph p = setParagraphUsingMessage(note, fromMessage,
paragraphId, text, title, params, config);
- if (!persistAndExecuteSingleParagraph(conn, note, p, true)) {
+ if (p.isEnabled() && !persistAndExecuteSingleParagraph(conn, note, p,
true)) {
// stop execution when one paragraph fails.
break;
}
zeppelin git commit: ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE.
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 a6189e9e6 -> ccaa0de7b
ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE.
What is this PR for?
During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and
other functions leaving unusable notebooks.
What type of PR is it?
[Bug Fix]
Todos
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3309
How should this be tested?
Manually tested using Import/Clone of Notebooks and attempt to adjust bound
interpreters
Screenshots (if appropriate)
Questions:
Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No
Author: Greg Senia gseniaapache.org
Author: gss2002
Closes #2855 from gss2002/ZEPPELIN-3309 and squashes the following commits:
b8c35c9 [gss2002] ZEPPELIN-3309. Import/Clone user not set in Paragraph causes
NPE.
(cherry picked from commit 803116fd0d4ad5baaf3d85b7c903d75fd04b2030)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/ccaa0de7
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/ccaa0de7
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/ccaa0de7
Branch: refs/heads/branch-0.8
Commit: ccaa0de7b9fe838cd5ce7047f9d8130ba8f23f38
Parents: a6189e9
Author: gss2002
Authored: Thu Mar 8 17:06:06 2018 -0500
Committer: Jeff Zhang
Committed: Sun Mar 11 07:18:48 2018 +0800
--
.../java/org/apache/zeppelin/notebook/Note.java | 8 ++-
.../org/apache/zeppelin/notebook/Notebook.java | 4 ++--
.../apache/zeppelin/notebook/NotebookTest.java | 23 +++-
3 files changed, 31 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/ccaa0de7/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 495f670..fc70c70 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -362,7 +362,7 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
*
* @param srcParagraph source paragraph
*/
- void addCloneParagraph(Paragraph srcParagraph) {
+ void addCloneParagraph(Paragraph srcParagraph, AuthenticationInfo subject) {
// Keep paragraph original ID
final Paragraph newParagraph = new Paragraph(srcParagraph.getId(), this,
this, factory);
@@ -371,11 +371,17 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
Map param = srcParagraph.settings.getParams();
LinkedHashMap form = srcParagraph.settings.getForms();
+logger.debug("srcParagraph user: " + srcParagraph.getUser());
+
+newParagraph.setAuthenticationInfo(subject);
newParagraph.setConfig(config);
newParagraph.settings.setParams(param);
newParagraph.settings.setForms(form);
newParagraph.setText(srcParagraph.getText());
newParagraph.setTitle(srcParagraph.getTitle());
+
+logger.debug("newParagraph user: " + newParagraph.getUser());
+
try {
Gson gson = new Gson();
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/ccaa0de7/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
index 64301e6..f4214e3 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
@@ -211,7 +211,7 @@ public class Notebook implements NoteEventListener {
newNote.setCronSupported(getConf());
List paragraphs = oldNote.getParagraphs();
for (Paragraph p : paragraphs) {
-newNote.addCloneParagraph(p);
+newNote.addCloneParagraph(p, subject);
}
notebookAuthorization.setNewNotePermissions(newNote.getId(), subject);
@@ -252,7 +252,7 @@ public class Notebook implements NoteEventListener {
List paragraphs = sourceNote.getParagraphs();
for (Paragraph p : paragraphs) {
- newNote.addCloneParagraph(p);
+ newNote.addCloneParagraph(p, subject);
}
noteSearchService.addIndexDoc(newNote);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/ccaa0de7/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
--
diff --git
zeppelin git commit: ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE.
Repository: zeppelin
Updated Branches:
refs/heads/master f4a798b62 -> 803116fd0
ZEPPELIN-3309. Import/Clone user not set in Paragraph causes NPE.
What is this PR for?
During Import/Clone Paragraph set "user" to eliminate NPEs thrown in Helium and
other functions leaving unusable notebooks.
What type of PR is it?
[Bug Fix]
Todos
What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3309
How should this be tested?
Manually tested using Import/Clone of Notebooks and attempt to adjust bound
interpreters
Screenshots (if appropriate)
Questions:
Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No
Author: Greg Senia gseniaapache.org
Author: gss2002
Closes #2855 from gss2002/ZEPPELIN-3309 and squashes the following commits:
b8c35c9 [gss2002] ZEPPELIN-3309. Import/Clone user not set in Paragraph causes
NPE.
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/803116fd
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/803116fd
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/803116fd
Branch: refs/heads/master
Commit: 803116fd0d4ad5baaf3d85b7c903d75fd04b2030
Parents: f4a798b
Author: gss2002
Authored: Thu Mar 8 17:06:06 2018 -0500
Committer: Jeff Zhang
Committed: Sun Mar 11 07:18:35 2018 +0800
--
.../java/org/apache/zeppelin/notebook/Note.java | 8 ++-
.../org/apache/zeppelin/notebook/Notebook.java | 4 ++--
.../apache/zeppelin/notebook/NotebookTest.java | 23 +++-
3 files changed, 31 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/803116fd/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 495f670..fc70c70 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -362,7 +362,7 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
*
* @param srcParagraph source paragraph
*/
- void addCloneParagraph(Paragraph srcParagraph) {
+ void addCloneParagraph(Paragraph srcParagraph, AuthenticationInfo subject) {
// Keep paragraph original ID
final Paragraph newParagraph = new Paragraph(srcParagraph.getId(), this,
this, factory);
@@ -371,11 +371,17 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
Map param = srcParagraph.settings.getParams();
LinkedHashMap form = srcParagraph.settings.getForms();
+logger.debug("srcParagraph user: " + srcParagraph.getUser());
+
+newParagraph.setAuthenticationInfo(subject);
newParagraph.setConfig(config);
newParagraph.settings.setParams(param);
newParagraph.settings.setForms(form);
newParagraph.setText(srcParagraph.getText());
newParagraph.setTitle(srcParagraph.getTitle());
+
+logger.debug("newParagraph user: " + newParagraph.getUser());
+
try {
Gson gson = new Gson();
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/803116fd/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
index a9168a3..7467e31 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java
@@ -211,7 +211,7 @@ public class Notebook implements NoteEventListener {
newNote.setCronSupported(getConf());
List paragraphs = oldNote.getParagraphs();
for (Paragraph p : paragraphs) {
-newNote.addCloneParagraph(p);
+newNote.addCloneParagraph(p, subject);
}
notebookAuthorization.setNewNotePermissions(newNote.getId(), subject);
@@ -252,7 +252,7 @@ public class Notebook implements NoteEventListener {
List paragraphs = sourceNote.getParagraphs();
for (Paragraph p : paragraphs) {
- newNote.addCloneParagraph(p);
+ newNote.addCloneParagraph(p, subject);
}
noteSearchService.addIndexDoc(newNote);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/803116fd/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
b/zeppelin-zengine/src
zeppelin git commit: ZEPPELIN-3322. Update interpreterBind when restarting zeppelin server
Repository: zeppelin
Updated Branches:
refs/heads/master 803116fd0 -> 4463967a5
ZEPPELIN-3322. Update interpreterBind when restarting zeppelin server
### What is this PR for?
When loading interpreter.json, also update interpreterBindings as we change
change interpreter setting id in ZEPPELIN-3208.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3322
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2861 from zjffdu/ZEPPELIN-3322 and squashes the following commits:
94f4882 [Jeff Zhang] ZEPPELIN-3322. Update interpreterBind when restarting
zeppelin server
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/4463967a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/4463967a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/4463967a
Branch: refs/heads/master
Commit: 4463967a520af2bc9950fdbd2987ed220d8c0344
Parents: 803116f
Author: Jeff Zhang
Authored: Tue Mar 13 14:54:58 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 13 16:03:15 2018 +0800
--
.../interpreter/InterpreterSettingManager.java | 15 --
.../InterpreterSettingManagerTest.java | 6 ++
.../src/test/resources/conf/interpreter.json| 21 +---
3 files changed, 20 insertions(+), 22 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4463967a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index c3ec81b..9587a88 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -208,6 +208,19 @@ public class InterpreterSettingManager {
return;
}
+// update interpreter binding first as we change interpreter setting id in
ZEPPELIN-3208.
+Map> newBindingMap = new HashMap<>();
+for (Map.Entry> entry :
infoSaving.interpreterBindings.entrySet()) {
+ String noteId = entry.getKey();
+ List oldSettingIdList = entry.getValue();
+ List newSettingIdList = new ArrayList<>();
+ for (String oldId : oldSettingIdList) {
+
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+ }
+ newBindingMap.put(noteId, newSettingIdList);
+}
+interpreterBindings.putAll(newBindingMap);
+
//TODO(zjffdu) still ugly (should move all to InterpreterInfoSaving)
for (InterpreterSetting savedInterpreterSetting :
infoSaving.interpreterSettings.values()) {
savedInterpreterSetting.setProperties(InterpreterSetting.convertInterpreterProperties(
@@ -259,8 +272,6 @@ public class InterpreterSettingManager {
interpreterSettings.put(savedInterpreterSetting.getId(),
savedInterpreterSetting);
}
-interpreterBindings.putAll(infoSaving.interpreterBindings);
-
if (infoSaving.interpreterRepositories != null) {
for (RemoteRepository repo : infoSaving.interpreterRepositories) {
if (!dependencyResolver.getRepos().contains(repo)) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4463967a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
index a52601c..10dfb03 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
@@ -72,6 +72,12 @@ public class InterpreterSettingManagerTest extends
AbstractInterpreterTest {
assertEquals(2, repositories.size());
assertEquals("central", repositories.get(0).getId());
+// verify interpreter binding
+List interpreterSettingIds =
interpreterSettingManager.getInterpreterBinding("2C6793KRV");
+assertEquals(2, interpreterSettingIds.size());
+assertEquals("test", interpreterSettingIds.get(0));
+
zeppelin git commit: ZEPPELIN-3322. Update interpreterBind when restarting zeppelin server
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 ccaa0de7b -> 7bae5ebb7
ZEPPELIN-3322. Update interpreterBind when restarting zeppelin server
### What is this PR for?
When loading interpreter.json, also update interpreterBindings as we change
change interpreter setting id in ZEPPELIN-3208.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3322
### How should this be tested?
* Unit test is added.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2861 from zjffdu/ZEPPELIN-3322 and squashes the following commits:
94f4882 [Jeff Zhang] ZEPPELIN-3322. Update interpreterBind when restarting
zeppelin server
(cherry picked from commit 4463967a520af2bc9950fdbd2987ed220d8c0344)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7bae5ebb
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7bae5ebb
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7bae5ebb
Branch: refs/heads/branch-0.8
Commit: 7bae5ebb790a96493e23000ca888e6adbb5c15a9
Parents: ccaa0de
Author: Jeff Zhang
Authored: Tue Mar 13 14:54:58 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 13 16:03:27 2018 +0800
--
.../interpreter/InterpreterSettingManager.java | 15 --
.../InterpreterSettingManagerTest.java | 6 ++
.../src/test/resources/conf/interpreter.json| 21 +---
3 files changed, 20 insertions(+), 22 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7bae5ebb/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index c3ec81b..9587a88 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -208,6 +208,19 @@ public class InterpreterSettingManager {
return;
}
+// update interpreter binding first as we change interpreter setting id in
ZEPPELIN-3208.
+Map> newBindingMap = new HashMap<>();
+for (Map.Entry> entry :
infoSaving.interpreterBindings.entrySet()) {
+ String noteId = entry.getKey();
+ List oldSettingIdList = entry.getValue();
+ List newSettingIdList = new ArrayList<>();
+ for (String oldId : oldSettingIdList) {
+
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+ }
+ newBindingMap.put(noteId, newSettingIdList);
+}
+interpreterBindings.putAll(newBindingMap);
+
//TODO(zjffdu) still ugly (should move all to InterpreterInfoSaving)
for (InterpreterSetting savedInterpreterSetting :
infoSaving.interpreterSettings.values()) {
savedInterpreterSetting.setProperties(InterpreterSetting.convertInterpreterProperties(
@@ -259,8 +272,6 @@ public class InterpreterSettingManager {
interpreterSettings.put(savedInterpreterSetting.getId(),
savedInterpreterSetting);
}
-interpreterBindings.putAll(infoSaving.interpreterBindings);
-
if (infoSaving.interpreterRepositories != null) {
for (RemoteRepository repo : infoSaving.interpreterRepositories) {
if (!dependencyResolver.getRepos().contains(repo)) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7bae5ebb/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
index a52601c..10dfb03 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/interpreter/InterpreterSettingManagerTest.java
@@ -72,6 +72,12 @@ public class InterpreterSettingManagerTest extends
AbstractInterpreterTest {
assertEquals(2, repositories.size());
assertEquals("central", repositories.get(0).getId());
+// verify interpreter binding
+List interpreterSettingIds =
interpreterSettingManager.getInterpreterBinding("2C6793KRV");
+a
zeppelin git commit: ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character - Encoding Issue
Repository: zeppelin
Updated Branches:
refs/heads/master 4463967a5 -> b9d2f40bd
ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character -
Encoding Issue
### What is this PR for?
ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character -
Encoding Issue
### What type of PR is it?
BUG FIX for : ZEPPELIN-3099
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3099
### How should this be tested?
test with German Special Characters like Ã, ü, Ã, ä, Ã, ö, à .
Author: Akhil Subhash Naik
Closes #2860 from Akhilsnaik/ZEPPELIN-3099-master and squashes the following
commits:
eee2f5a [Akhil Subhash Naik] ZEPPELIN-3099 Implement the UT cases and fix
review comments (asnaik)
ecf4b54 [Akhil Subhash Naik] ZEPPELIN-3099 Implement the UT cases and fix
review comments (asnaik)
7a65776 [Akhil Subhash Naik] ZEPPELIN-3099 Livy Interpreter doesn't support
German Special Character - Encoding Issue (asnaik)
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b9d2f40b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b9d2f40b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b9d2f40b
Branch: refs/heads/master
Commit: b9d2f40bd737163be55681970b1f467f668c8fab
Parents: 4463967
Author: Akhil Subhash Naik
Authored: Tue Mar 13 16:10:07 2018 +0530
Committer: Jeff Zhang
Committed: Tue Mar 13 19:55:35 2018 +0800
--
.../apache/zeppelin/livy/BaseLivyInterpreter.java | 15 ++-
.../apache/zeppelin/livy/LivyInterpreterIT.java | 18 ++
2 files changed, 20 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b9d2f40b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
--
diff --git
a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
index 5fe7ce4..1eba9c2 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
@@ -19,6 +19,7 @@ package org.apache.zeppelin.livy;
import java.io.FileInputStream;
import java.io.IOException;
+import java.nio.charset.Charset;
import java.security.KeyStore;
import java.security.Principal;
import java.util.ArrayList;
@@ -52,6 +53,7 @@ import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterResult;
@@ -68,6 +70,7 @@ import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
+import org.springframework.http.converter.StringHttpMessageConverter;
import org.springframework.security.kerberos.client.KerberosRestTemplate;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.HttpServerErrorException;
@@ -652,19 +655,21 @@ public abstract class BaseLivyInterpreter extends
Interpreter {
}
}
-
+RestTemplate restTemplate = null;
if (isSpnegoEnabled) {
if (httpClient == null) {
-return new KerberosRestTemplate(keytabLocation, principal);
+restTemplate = new KerberosRestTemplate(keytabLocation, principal);
} else {
-return new KerberosRestTemplate(keytabLocation, principal, httpClient);
+restTemplate = new KerberosRestTemplate(keytabLocation, principal,
httpClient);
}
}
if (httpClient == null) {
- return new RestTemplate();
+ restTemplate = new RestTemplate();
} else {
- return new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
}
+restTemplate.getMessageConverters().add(0, new
StringHttpMessageConverter(Charset.forName("UTF-8")));
+return restTemplate;
}
private String callRestAPI(String targetURL, String method) throws
LivyException {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b9d2f40b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
--
diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
b/livy/src/test/java/org/apach
zeppelin git commit: ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character - Encoding Issue
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 7bae5ebb7 -> a30d190bd
ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character -
Encoding Issue
### What is this PR for?
ZEPPELIN-3099 Livy Interpreter doesn't support German Special Character -
Encoding Issue
### What type of PR is it?
BUG FIX for : ZEPPELIN-3099
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3099
### How should this be tested?
test with German Special Characters like Ã, ü, Ã, ä, Ã, ö, à .
Author: Akhil Subhash Naik
Closes #2860 from Akhilsnaik/ZEPPELIN-3099-master and squashes the following
commits:
eee2f5a [Akhil Subhash Naik] ZEPPELIN-3099 Implement the UT cases and fix
review comments (asnaik)
ecf4b54 [Akhil Subhash Naik] ZEPPELIN-3099 Implement the UT cases and fix
review comments (asnaik)
7a65776 [Akhil Subhash Naik] ZEPPELIN-3099 Livy Interpreter doesn't support
German Special Character - Encoding Issue (asnaik)
(cherry picked from commit b9d2f40bd737163be55681970b1f467f668c8fab)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a30d190b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a30d190b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a30d190b
Branch: refs/heads/branch-0.8
Commit: a30d190bdc6db0fb37a02b5f29eecaec575f2733
Parents: 7bae5eb
Author: Akhil Subhash Naik
Authored: Tue Mar 13 16:10:07 2018 +0530
Committer: Jeff Zhang
Committed: Tue Mar 13 19:55:52 2018 +0800
--
.../apache/zeppelin/livy/BaseLivyInterpreter.java | 15 ++-
.../apache/zeppelin/livy/LivyInterpreterIT.java | 18 ++
2 files changed, 20 insertions(+), 13 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a30d190b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
--
diff --git
a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
index 5fe7ce4..1eba9c2 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
@@ -19,6 +19,7 @@ package org.apache.zeppelin.livy;
import java.io.FileInputStream;
import java.io.IOException;
+import java.nio.charset.Charset;
import java.security.KeyStore;
import java.security.Principal;
import java.util.ArrayList;
@@ -52,6 +53,7 @@ import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.client.HttpClients;
import org.apache.zeppelin.interpreter.Interpreter;
+import org.apache.zeppelin.interpreter.Interpreter.FormType;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterResult;
@@ -68,6 +70,7 @@ import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
+import org.springframework.http.converter.StringHttpMessageConverter;
import org.springframework.security.kerberos.client.KerberosRestTemplate;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.HttpServerErrorException;
@@ -652,19 +655,21 @@ public abstract class BaseLivyInterpreter extends
Interpreter {
}
}
-
+RestTemplate restTemplate = null;
if (isSpnegoEnabled) {
if (httpClient == null) {
-return new KerberosRestTemplate(keytabLocation, principal);
+restTemplate = new KerberosRestTemplate(keytabLocation, principal);
} else {
-return new KerberosRestTemplate(keytabLocation, principal, httpClient);
+restTemplate = new KerberosRestTemplate(keytabLocation, principal,
httpClient);
}
}
if (httpClient == null) {
- return new RestTemplate();
+ restTemplate = new RestTemplate();
} else {
- return new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
}
+restTemplate.getMessageConverters().add(0, new
StringHttpMessageConverter(Charset.forName("UTF-8")));
+return restTemplate;
}
private String callRestAPI(String targetURL, String method) throws
LivyException {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a30d190b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
--
diff -
zeppelin git commit: ZEPPELIN-3159. Fixed Checkstyle errors and warnings in jdbc module
Repository: zeppelin Updated Branches: refs/heads/master b9d2f40bd -> 6919c4ad3 ZEPPELIN-3159. Fixed Checkstyle errors and warnings in jdbc module ### What is this PR for? Fixed the Checkstyle errors and warnings in the jdbc module. ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3159 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Jan Hentschel Closes #2801 from HorizonNet/ZEPPELIN-3159 and squashes the following commits: 7771c26 [Jan Hentschel] ZEPPELIN-3159. Fixed Checkstyle errors and warnings in jdbc module Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/6919c4ad Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/6919c4ad Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/6919c4ad Branch: refs/heads/master Commit: 6919c4ad32e872f3658506c74447ab353d89c89e Parents: b9d2f40 Author: Jan Hentschel Authored: Wed Feb 14 14:38:58 2018 +0100 Committer: Jeff Zhang Committed: Wed Mar 14 09:59:53 2018 +0800 -- jdbc/pom.xml| 7 + .../apache/zeppelin/jdbc/JDBCInterpreter.java | 65 --- .../zeppelin/jdbc/JDBCUserConfigurations.java | 5 +- .../org/apache/zeppelin/jdbc/SqlCompleter.java | 58 +++ .../jdbc/security/JDBCSecurityImpl.java | 12 +- .../zeppelin/jdbc/JDBCInterpreterTest.java | 74 .../apache/zeppelin/jdbc/SqlCompleterTest.java | 171 +++ 7 files changed, 214 insertions(+), 178 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/6919c4ad/jdbc/pom.xml -- diff --git a/jdbc/pom.xml b/jdbc/pom.xml index ba745dd..eef9aa8 100644 --- a/jdbc/pom.xml +++ b/jdbc/pom.xml @@ -362,6 +362,13 @@ maven-resources-plugin + +org.apache.maven.plugins +maven-checkstyle-plugin + + false + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/6919c4ad/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java -- diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java index 0265e2d..56afe6f 100644 --- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java +++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java @@ -14,6 +14,27 @@ */ package org.apache.zeppelin.jdbc; +import static org.apache.commons.lang.StringUtils.containsIgnoreCase; +import static org.apache.commons.lang.StringUtils.isEmpty; +import static org.apache.commons.lang.StringUtils.isNotEmpty; +import static org.apache.hadoop.security.UserGroupInformation.AuthenticationMethod.KERBEROS; + +import org.apache.commons.dbcp2.ConnectionFactory; +import org.apache.commons.dbcp2.DriverManagerConnectionFactory; +import org.apache.commons.dbcp2.PoolableConnectionFactory; +import org.apache.commons.dbcp2.PoolingDriver; +import org.apache.commons.lang.StringUtils; +import org.apache.commons.lang.exception.ExceptionUtils; +import org.apache.commons.lang.mutable.MutableBoolean; +import org.apache.commons.pool2.ObjectPool; +import org.apache.commons.pool2.impl.GenericObjectPool; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.alias.CredentialProvider; +import org.apache.hadoop.security.alias.CredentialProviderFactory; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + import java.io.IOException; import java.security.PrivilegedExceptionAction; import java.sql.Connection; @@ -33,19 +54,6 @@ import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; -import org.apache.commons.dbcp2.ConnectionFactory; -import org.apache.commons.dbcp2.DriverManagerConnectionFactory; -import org.apache.commons.dbcp2.PoolableConnectionFactory; -import org.apache.commons.dbcp2.PoolingDriver; -import org.apache.commons.lang.StringUtils; -import org.apache.commons.lang.exception.ExceptionUtils; -import org.apache.commons.lang.mutable.MutableBoolean; -import org.apache.commons.pool2.ObjectPool; -import org.apache.commons.pool2.impl.GenericObjectPool; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.security.UserGroupInformation; -import org.apache.hadoop.security.alias.CredentialProvider; -import org.apache.hadoop.security.alias.CredentialProvider
zeppelin git commit: ZEPPELIN-3158. Fixed Checkstyle errors and warnings in ignite module
Repository: zeppelin Updated Branches: refs/heads/master 6919c4ad3 -> 5e37eed12 ZEPPELIN-3158. Fixed Checkstyle errors and warnings in ignite module ### What is this PR for? Fixed all Checkstyle errors and warnings in the ignite module. ### What type of PR is it? Improvement ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3158 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? no * Is there breaking changes for older versions? no * Does this needs documentation? no Author: Jan Hentschel Closes #2797 from HorizonNet/ZEPPELIN-3158 and squashes the following commits: a4bb15b [Jan Hentschel] ZEPPELIN-3158. Fixed Checkstyle errors and warnings in ignite module Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/5e37eed1 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/5e37eed1 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/5e37eed1 Branch: refs/heads/master Commit: 5e37eed1270b1aa4a2ed9f540fee16fad29127b4 Parents: 6919c4a Author: Jan Hentschel Authored: Tue Feb 13 14:59:53 2018 +0100 Committer: Jeff Zhang Committed: Wed Mar 14 10:01:03 2018 +0800 -- ignite/pom.xml | 7 ++ .../zeppelin/ignite/IgniteInterpreter.java | 17 -- .../zeppelin/ignite/IgniteSqlInterpreter.java | 20 .../zeppelin/ignite/IgniteInterpreterTest.java | 23 +++ .../ignite/IgniteSqlInterpreterTest.java| 24 .../java/org/apache/zeppelin/ignite/Person.java | 4 ++-- 6 files changed, 56 insertions(+), 39 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5e37eed1/ignite/pom.xml -- diff --git a/ignite/pom.xml b/ignite/pom.xml index cd778b8..7d7b3db 100644 --- a/ignite/pom.xml +++ b/ignite/pom.xml @@ -114,6 +114,13 @@ maven-resources-plugin + +org.apache.maven.plugins +maven-checkstyle-plugin + + false + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5e37eed1/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteInterpreter.java -- diff --git a/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteInterpreter.java b/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteInterpreter.java index ac385ea..e104039 100644 --- a/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteInterpreter.java +++ b/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteInterpreter.java @@ -21,11 +21,6 @@ import org.apache.ignite.Ignition; import org.apache.ignite.configuration.IgniteConfiguration; import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; -import org.apache.zeppelin.interpreter.*; -import org.apache.zeppelin.interpreter.InterpreterResult.Code; -import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion; -import org.apache.zeppelin.scheduler.Scheduler; -import org.apache.zeppelin.scheduler.SchedulerFactory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; @@ -36,13 +31,12 @@ import java.net.URL; import java.net.URLClassLoader; import java.util.ArrayList; import java.util.Collections; -import java.util.List; import java.util.LinkedList; +import java.util.List; import java.util.Map; import java.util.Properties; import scala.Console; -import scala.None; import scala.Some; import scala.collection.JavaConversions; import scala.tools.nsc.Settings; @@ -51,6 +45,15 @@ import scala.tools.nsc.interpreter.Results.Result; import scala.tools.nsc.settings.MutableSettings.BooleanSetting; import scala.tools.nsc.settings.MutableSettings.PathSetting; +import org.apache.zeppelin.interpreter.Interpreter; +import org.apache.zeppelin.interpreter.InterpreterContext; +import org.apache.zeppelin.interpreter.InterpreterResult; +import org.apache.zeppelin.interpreter.InterpreterResult.Code; +import org.apache.zeppelin.interpreter.InterpreterUtils; +import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion; +import org.apache.zeppelin.scheduler.Scheduler; +import org.apache.zeppelin.scheduler.SchedulerFactory; + /** * Apache Ignite interpreter (http://ignite.incubator.apache.org/). * http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5e37eed1/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteSqlInterpreter.java -- diff --git a/ignite/src/main/java/org/apache/zeppelin/ignite/IgniteSqlInterpreter.java b/ignite/src/main/java/org
zeppelin git commit: [HOTFIX] More proper error message when interpreter is not binded
Repository: zeppelin
Updated Branches:
refs/heads/master 5e37eed12 -> 514ba659b
[HOTFIX] More proper error message when interpreter is not binded
### What is this PR for?
Minor fix for more proper error message when interpreter is not binded
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* NO jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2865 from zjffdu/HOTFIX_INTP and squashes the following commits:
0a22913 [Jeff Zhang] [HOTFIX] More proper error message when interpreter is not
binded
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/514ba659
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/514ba659
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/514ba659
Branch: refs/heads/master
Commit: 514ba659bbe8706bccf539b09712362377bfd588
Parents: 5e37eed
Author: Jeff Zhang
Authored: Wed Mar 14 10:53:28 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 12:26:35 2018 +0800
--
.../org/apache/zeppelin/interpreter/InterpreterSetting.java | 5 +++--
.../main/java/org/apache/zeppelin/notebook/Paragraph.java | 9 +++--
2 files changed, 6 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/514ba659/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index ff4626d..a3b7f8d 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -677,8 +677,9 @@ public class InterpreterSetting {
List createInterpreters(String user, String interpreterGroupId,
String sessionId) {
List interpreters = new ArrayList<>();
List interpreterInfos = getInterpreterInfos();
+Properties intpProperties = getJavaProperties();
for (InterpreterInfo info : interpreterInfos) {
- Interpreter interpreter = new RemoteInterpreter(getJavaProperties(),
sessionId,
+ Interpreter interpreter = new RemoteInterpreter(intpProperties,
sessionId,
info.getClassName(), user, lifecycleManager);
if (info.isDefaultInterpreter()) {
interpreters.add(0, interpreter);
@@ -688,7 +689,7 @@ public class InterpreterSetting {
LOGGER.info("Interpreter {} created for user: {}, sessionId: {}",
interpreter.getClassName(), user, sessionId);
}
-interpreters.add(new ConfInterpreter(getJavaProperties(),
interpreterGroupId, this));
+interpreters.add(new ConfInterpreter(intpProperties, interpreterGroupId,
this));
return interpreters;
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/514ba659/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
index 23d2d47..57756b8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
@@ -367,14 +367,11 @@ public class Paragraph extends Job implements Cloneable,
JsonSerializable {
return true;
}
} catch (InterpreterNotFoundException e) {
- String intpExceptionMsg =
- getJobName() + "'s Interpreter " + getIntpText() + " not found";
- RuntimeException intpException = new RuntimeException(intpExceptionMsg);
InterpreterResult intpResult =
- new InterpreterResult(InterpreterResult.Code.ERROR,
intpException.getMessage());
- setReturn(intpResult, intpException);
+ new InterpreterResult(InterpreterResult.Code.ERROR);
+ setReturn(intpResult, e);
setStatus(Job.Status.ERROR);
- throw intpException;
+ throw new RuntimeException(e);
}
}
zeppelin git commit: [HOTFIX] More proper error message when interpreter is not binded
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 a30d190bd -> 28dc47da2
[HOTFIX] More proper error message when interpreter is not binded
### What is this PR for?
Minor fix for more proper error message when interpreter is not binded
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* NO jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
Before
After
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2865 from zjffdu/HOTFIX_INTP and squashes the following commits:
0a22913 [Jeff Zhang] [HOTFIX] More proper error message when interpreter is not
binded
(cherry picked from commit 514ba659bbe8706bccf539b09712362377bfd588)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/28dc47da
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/28dc47da
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/28dc47da
Branch: refs/heads/branch-0.8
Commit: 28dc47da28a7b649193ff052e5bb9941855943c3
Parents: a30d190
Author: Jeff Zhang
Authored: Wed Mar 14 10:53:28 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 12:26:50 2018 +0800
--
.../org/apache/zeppelin/interpreter/InterpreterSetting.java | 5 +++--
.../main/java/org/apache/zeppelin/notebook/Paragraph.java | 9 +++--
2 files changed, 6 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/28dc47da/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index 6e51a52..b63c98a 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -676,8 +676,9 @@ public class InterpreterSetting {
List createInterpreters(String user, String interpreterGroupId,
String sessionId) {
List interpreters = new ArrayList<>();
List interpreterInfos = getInterpreterInfos();
+Properties intpProperties = getJavaProperties();
for (InterpreterInfo info : interpreterInfos) {
- Interpreter interpreter = new RemoteInterpreter(getJavaProperties(),
sessionId,
+ Interpreter interpreter = new RemoteInterpreter(intpProperties,
sessionId,
info.getClassName(), user, lifecycleManager);
if (info.isDefaultInterpreter()) {
interpreters.add(0, interpreter);
@@ -687,7 +688,7 @@ public class InterpreterSetting {
LOGGER.info("Interpreter {} created for user: {}, sessionId: {}",
interpreter.getClassName(), user, sessionId);
}
-interpreters.add(new ConfInterpreter(getJavaProperties(),
interpreterGroupId, this));
+interpreters.add(new ConfInterpreter(intpProperties, interpreterGroupId,
this));
return interpreters;
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/28dc47da/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
index 23d2d47..57756b8 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Paragraph.java
@@ -367,14 +367,11 @@ public class Paragraph extends Job implements Cloneable,
JsonSerializable {
return true;
}
} catch (InterpreterNotFoundException e) {
- String intpExceptionMsg =
- getJobName() + "'s Interpreter " + getIntpText() + " not found";
- RuntimeException intpException = new RuntimeException(intpExceptionMsg);
InterpreterResult intpResult =
- new InterpreterResult(InterpreterResult.Code.ERROR,
intpException.getMessage());
- setReturn(intpResult, intpException);
+ new InterpreterResult(InterpreterResult.Code.ERROR);
+ setReturn(intpResult, e);
setStatus(Job.Status.ERROR);
zeppelin git commit: ZEPPELIN-3328. Add plotting test for LivyInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master 514ba659b -> 9ec6eb5a4
ZEPPELIN-3328. Add plotting test for LivyInterpreter
### What is this PR for?
Add plotting for LivyInterpreter (Python & R)
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3328
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2866 from zjffdu/ZEPPELIN-3328 and squashes the following commits:
fd394e3 [Jeff Zhang] ZEPPELIN-3328. Add plotting test for LivyInterpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/9ec6eb5a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/9ec6eb5a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/9ec6eb5a
Branch: refs/heads/master
Commit: 9ec6eb5a4e422e4ac73a97e38e47512811d5b29d
Parents: 514ba65
Author: Jeff Zhang
Authored: Wed Mar 14 13:27:09 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 14:44:06 2018 +0800
--
.../apache/zeppelin/livy/LivyInterpreterIT.java | 20
1 file changed, 20 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/9ec6eb5a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
--
diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
index 0c8bea9..c7fbc8d 100644
--- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
+++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
@@ -750,6 +750,26 @@ public class LivyInterpreterIT {
assertEquals(1, result.message().size());
assertTrue(result.message().get(0).getData().contains("col_1 col_2\n1
hello20"));
}
+
+ // test plotting of python
+ result = pysparkInterpreter.interpret(
+ "import matplotlib.pyplot as plt\n" +
+ "plt.switch_backend('agg')\n" +
+ "data=[1,2,3,4]\n" +
+ "plt.figure()\n" +
+ "plt.plot(data)\n" +
+ "%matplot plt", context);
+ assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+ assertEquals(1, result.message().size());
+ assertEquals(InterpreterResult.Type.IMG,
result.message().get(0).getType());
+
+ // test plotting of R
+ result = sparkRInterpreter.interpret(
+ "hist(mtcars$mpg)", context);
+ assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+ assertEquals(1, result.message().size());
+ assertEquals(InterpreterResult.Type.IMG,
result.message().get(0).getType());
+
} finally {
sparkInterpreter.close();
sqlInterpreter.close();
zeppelin git commit: ZEPPELIN-3328. Add plotting test for LivyInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 28dc47da2 -> 0211590ab
ZEPPELIN-3328. Add plotting test for LivyInterpreter
### What is this PR for?
Add plotting for LivyInterpreter (Python & R)
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3328
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2866 from zjffdu/ZEPPELIN-3328 and squashes the following commits:
fd394e3 [Jeff Zhang] ZEPPELIN-3328. Add plotting test for LivyInterpreter
(cherry picked from commit 9ec6eb5a4e422e4ac73a97e38e47512811d5b29d)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/0211590a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/0211590a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/0211590a
Branch: refs/heads/branch-0.8
Commit: 0211590ab5ea8fce2dbcc0de991fcf895170360e
Parents: 28dc47d
Author: Jeff Zhang
Authored: Wed Mar 14 13:27:09 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 14:44:26 2018 +0800
--
.../apache/zeppelin/livy/LivyInterpreterIT.java | 20
1 file changed, 20 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0211590a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
--
diff --git a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
index 0c8bea9..c7fbc8d 100644
--- a/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
+++ b/livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java
@@ -750,6 +750,26 @@ public class LivyInterpreterIT {
assertEquals(1, result.message().size());
assertTrue(result.message().get(0).getData().contains("col_1 col_2\n1
hello20"));
}
+
+ // test plotting of python
+ result = pysparkInterpreter.interpret(
+ "import matplotlib.pyplot as plt\n" +
+ "plt.switch_backend('agg')\n" +
+ "data=[1,2,3,4]\n" +
+ "plt.figure()\n" +
+ "plt.plot(data)\n" +
+ "%matplot plt", context);
+ assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+ assertEquals(1, result.message().size());
+ assertEquals(InterpreterResult.Type.IMG,
result.message().get(0).getType());
+
+ // test plotting of R
+ result = sparkRInterpreter.interpret(
+ "hist(mtcars$mpg)", context);
+ assertEquals(InterpreterResult.Code.SUCCESS, result.code());
+ assertEquals(1, result.message().size());
+ assertEquals(InterpreterResult.Type.IMG,
result.message().get(0).getType());
+
} finally {
sparkInterpreter.close();
sqlInterpreter.close();
zeppelin git commit: ZEPPELIN-3314. Only apply maven-dependency-plugin for interpreter component
Repository: zeppelin
Updated Branches:
refs/heads/master 9ec6eb5a4 -> 1effc6350
ZEPPELIN-3314. Only apply maven-dependency-plugin for interpreter component
### What is this PR for?
only interpreter component need to copy interpreter dependency and
interpreter-setting.json
### What type of PR is it?
[Bug Fix | Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3314
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2857 from zjffdu/ZEPPELIN-3314 and squashes the following commits:
322530f [Jeff Zhang] ZEPPELIN-3314. Only apply maven-dependency-plugin for
interpreter component
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/1effc635
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/1effc635
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/1effc635
Branch: refs/heads/master
Commit: 1effc63507f819f2b99589a34605afb53226282d
Parents: 9ec6eb5
Author: Jeff Zhang
Authored: Mon Mar 12 09:41:01 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 20:48:58 2018 +0800
--
interpreter-parent/pom.xml | 8
pom.xml | 36
zeppelin-integration/pom.xml | 4
zeppelin-interpreter/pom.xml | 18 +-
zeppelin-server/pom.xml | 6 ++
zeppelin-zengine/pom.xml | 5 +
6 files changed, 36 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/1effc635/interpreter-parent/pom.xml
--
diff --git a/interpreter-parent/pom.xml b/interpreter-parent/pom.xml
index cbfedd5..0aa2026 100644
--- a/interpreter-parent/pom.xml
+++ b/interpreter-parent/pom.xml
@@ -79,14 +79,6 @@
maven-dependency-plugin
- copy-dependencies
- none
-
-true
-
-
-
-
copy-interpreter-dependencies
package
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/1effc635/pom.xml
--
diff --git a/pom.xml b/pom.xml
index d6087a8..ca6c409 100644
--- a/pom.xml
+++ b/pom.xml
@@ -450,26 +450,6 @@
-maven-dependency-plugin
-
-
-copy-dependencies
-process-test-resources
-
- copy-dependencies
-
-
- ${project.build.directory}/lib
- false
- false
- true
- runtime
-
-
-
-
-
-
maven-jar-plugin
${plugin.jar.version}
@@ -589,6 +569,22 @@
maven-dependency-plugin
${plugin.dependency.version}
+
+
+ copy-dependencies
+ process-test-resources
+
+copy-dependencies
+
+
+
${project.build.directory}/lib
+false
+false
+true
+runtime
+
+
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/1effc635/zeppelin-integration/pom.xml
--
diff --git a/zeppelin-integration/pom.xml b/zeppelin-integration/pom.xml
index 5240a2c..94ed3a2 100644
--- a/zeppelin-integration/pom.xml
+++ b/zeppelin-integration/pom.xml
@@ -189,6 +189,10 @@
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/1effc635/zeppelin-interpreter/pom.xml
--
diff --git a/zeppelin-interpreter/pom.xml b/zeppelin-interpreter/pom.xml
index 80864f9..7fe75a7 100644
--- a/zeppelin-interpreter/pom.xml
+++ b/zeppelin-interpreter/pom.xml
@@ -247,20 +247,12 @@
false
+
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
-
-
-
- maven-enforcer-plugin
-
-
- enforce
- none
-
-
-
-
-
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/1effc635/zeppelin-server/pom.xml
--
diff --gi
zeppelin git commit: ZEPPELIN-3314. Only apply maven-dependency-plugin for interpreter component
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 0211590ab -> 48e1c4bba
ZEPPELIN-3314. Only apply maven-dependency-plugin for interpreter component
only interpreter component need to copy interpreter dependency and
interpreter-setting.json
[Bug Fix | Improvement ]
* [ ] - Task
* https://issues.apache.org/jira/browse/ZEPPELIN-3314
* CI pass
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2857 from zjffdu/ZEPPELIN-3314 and squashes the following commits:
322530f [Jeff Zhang] ZEPPELIN-3314. Only apply maven-dependency-plugin for
interpreter component
(cherry picked from commit 1effc63507f819f2b99589a34605afb53226282d)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/48e1c4bb
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/48e1c4bb
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/48e1c4bb
Branch: refs/heads/branch-0.8
Commit: 48e1c4bba95af6ff438302a823cb0f5b88ae5e4b
Parents: 0211590
Author: Jeff Zhang
Authored: Mon Mar 12 09:41:01 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 20:51:02 2018 +0800
--
interpreter-parent/pom.xml | 8
pom.xml | 36
zeppelin-integration/pom.xml | 4
zeppelin-interpreter/pom.xml | 27 ++-
zeppelin-server/pom.xml | 6 ++
zeppelin-zengine/pom.xml | 5 +
6 files changed, 45 insertions(+), 41 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/48e1c4bb/interpreter-parent/pom.xml
--
diff --git a/interpreter-parent/pom.xml b/interpreter-parent/pom.xml
index fc924c1..c40bc11 100644
--- a/interpreter-parent/pom.xml
+++ b/interpreter-parent/pom.xml
@@ -79,14 +79,6 @@
maven-dependency-plugin
- copy-dependencies
- none
-
-true
-
-
-
-
copy-interpreter-dependencies
package
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/48e1c4bb/pom.xml
--
diff --git a/pom.xml b/pom.xml
index efb1abd..592052e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -442,26 +442,6 @@
-maven-dependency-plugin
-
-
-copy-dependencies
-process-test-resources
-
- copy-dependencies
-
-
- ${project.build.directory}/lib
- false
- false
- true
- runtime
-
-
-
-
-
-
maven-jar-plugin
${plugin.jar.version}
@@ -581,6 +561,22 @@
maven-dependency-plugin
${plugin.dependency.version}
+
+
+ copy-dependencies
+ process-test-resources
+
+copy-dependencies
+
+
+
${project.build.directory}/lib
+false
+false
+true
+runtime
+
+
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/48e1c4bb/zeppelin-integration/pom.xml
--
diff --git a/zeppelin-integration/pom.xml b/zeppelin-integration/pom.xml
index 1a709f9..41f8833 100644
--- a/zeppelin-integration/pom.xml
+++ b/zeppelin-integration/pom.xml
@@ -189,6 +189,10 @@
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/48e1c4bb/zeppelin-interpreter/pom.xml
--
diff --git a/zeppelin-interpreter/pom.xml b/zeppelin-interpreter/pom.xml
index b0fd992..30f4991 100644
--- a/zeppelin-interpreter/pom.xml
+++ b/zeppelin-interpreter/pom.xml
@@ -239,19 +239,20 @@
-
-
-
- maven-enforcer-plugin
-
-
- enforce
- none
-
-
-
-
-
+
+
+org.apache.maven.plugins
+maven-checkstyle-plugin
+
+ false
+
+
+
+
+org.apache.maven.plugins
+maven-dependency-plugin
+
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/48e1c4bb/zeppelin-server/pom.
zeppelin git commit: ZEPPELIN-3315. Merge beforeStatusChange and afterStatusChange to onStatusChange
Repository: zeppelin
Updated Branches:
refs/heads/master 1effc6350 -> 4ee09717c
ZEPPELIN-3315. Merge beforeStatusChange and afterStatusChange to onStatusChange
### What is this PR for?
The signature of `beforeStatusChange` & `afterStatusChange` include both the
status of before and after, so it is not necessary to create both
`beforeStatusChange` & `afterStatusChange`. Only one method `onStatusChange` is
sufficient.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3315
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2858 from zjffdu/ZEPPELIN-3315 and squashes the following commits:
f0fe594 [Jeff Zhang] ZEPPELIN-3315. Merge beforeStatusChange and
afterStatusChange to onStatusChange
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/4ee09717
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/4ee09717
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/4ee09717
Branch: refs/heads/master
Commit: 4ee09717cfb2d09b55f20fbd8245b6c2eca37fda
Parents: 1effc63
Author: Jeff Zhang
Authored: Mon Mar 12 10:12:51 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 20:52:46 2018 +0800
--
.../interpreter/remote/RemoteInterpreterServer.java | 6 +-
.../java/org/apache/zeppelin/scheduler/Job.java | 7 ++-
.../org/apache/zeppelin/scheduler/JobListener.java | 6 ++
.../org/apache/zeppelin/socket/NotebookServer.java | 6 +-
.../java/org/apache/zeppelin/notebook/Note.java | 16 +++-
.../apache/zeppelin/scheduler/RemoteScheduler.java | 10 ++
.../helium/HeliumApplicationFactoryTest.java| 7 +--
.../org/apache/zeppelin/notebook/NotebookTest.java | 6 +-
.../notebook/repo/NotebookRepoSyncTest.java | 5 +
9 files changed, 14 insertions(+), 55 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4ee09717/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
index d50d0ed..766ed16 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/remote/RemoteInterpreterServer.java
@@ -513,11 +513,7 @@ public class RemoteInterpreterServer extends Thread
}
@Override
-public void beforeStatusChange(Job job, Status before, Status after) {
-}
-
-@Override
-public void afterStatusChange(Job job, Status before, Status after) {
+public void onStatusChange(Job job, Status before, Status after) {
synchronized (this) {
notifyAll();
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4ee09717/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/Job.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/Job.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/Job.java
index 8e25f7b..579b604 100644
--- a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/Job.java
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/Job.java
@@ -152,12 +152,9 @@ public abstract class Job {
}
Status before = this.status;
Status after = status;
-if (listener != null) {
- listener.beforeStatusChange(this, before, after);
-}
this.status = status;
-if (listener != null) {
- listener.afterStatusChange(this, before, after);
+if (listener != null && before != after) {
+ listener.onStatusChange(this, before, after);
}
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4ee09717/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/JobListener.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/JobListener.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/JobListener.java
index 3042941..00f8d81 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/scheduler/JobListener.java
+++
b/zeppelin-interpreter/src/main/java/org/apache
zeppelin git commit: ZEPPELIN-2515. After 100 minutes R process quits silently and spark.r interpreter becomes unresponsive
Repository: zeppelin
Updated Branches:
refs/heads/master 4ee09717c -> 8b2259cf5
ZEPPELIN-2515. After 100 minutes R process quits silently and spark.r
interpreter becomes unresponsive
### What is this PR for?
This PR would 2 features:
1. Make timeout of sparkr backend configurable.
2. Detect R backend dead and display proper message to frontend.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2515
### How should this be tested?
* unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2839 from zjffdu/ZEPPELIN-2515 and squashes the following commits:
62888c5 [Jeff Zhang] ZEPPELIN-2515. After 100 minutes R process quits silently
and spark.r interpreter becomes unresponsive
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/8b2259cf
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/8b2259cf
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/8b2259cf
Branch: refs/heads/master
Commit: 8b2259cf5805f654b5da2725fa8207fa0b6d5f7e
Parents: 4ee0971
Author: Jeff Zhang
Authored: Tue Mar 6 15:02:19 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 20:54:35 2018 +0800
--
.../zeppelin/spark/SparkRInterpreter.java | 17 +--
.../org/apache/zeppelin/spark/ZeppelinR.java| 32 ++--
.../src/main/resources/R/zeppelin_sparkr.R | 3 +-
.../zeppelin/spark/SparkRInterpreterTest.java | 9 +-
4 files changed, 54 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/8b2259cf/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
index 5efff0e..44f71b7 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
@@ -27,6 +27,7 @@ import org.apache.spark.SparkRBackend;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.zeppelin.interpreter.*;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
+import org.apache.zeppelin.interpreter.util.InterpreterOutputStream;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.slf4j.Logger;
@@ -36,6 +37,8 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
/**
* R and SparkR interpreter with visualization support.
@@ -46,6 +49,7 @@ public class SparkRInterpreter extends Interpreter {
private static String renderOptions;
private SparkInterpreter sparkInterpreter;
private ZeppelinR zeppelinR;
+ private AtomicBoolean rbackendDead = new AtomicBoolean(false);
private SparkContext sc;
private JavaSparkContext jsc;
@@ -79,10 +83,11 @@ public class SparkRInterpreter extends Interpreter {
}
int port = SparkRBackend.port();
-
this.sparkInterpreter = getSparkInterpreter();
this.sc = sparkInterpreter.getSparkContext();
this.jsc = sparkInterpreter.getJavaSparkContext();
+int timeout = this.sc.getConf().getInt("spark.r.backendConnectionTimeout",
6000);
+
SparkVersion sparkVersion = new SparkVersion(sc.version());
ZeppelinRContext.setSparkContext(sc);
ZeppelinRContext.setJavaSparkContext(jsc);
@@ -92,7 +97,7 @@ public class SparkRInterpreter extends Interpreter {
ZeppelinRContext.setSqlContext(sparkInterpreter.getSQLContext());
ZeppelinRContext.setZeppelinContext(sparkInterpreter.getZeppelinContext());
-zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port, sparkVersion);
+zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port, sparkVersion,
timeout, this);
try {
zeppelinR.open();
} catch (IOException e) {
@@ -159,6 +164,10 @@ public class SparkRInterpreter extends Interpreter {
try {
// render output with knitr
+ if (rbackendDead.get()) {
+return new InterpreterResult(InterpreterResult.Code.ERROR,
+"sparkR backend is dead, please try to increase
spark.r.backendConnectionTimeout");
+ }
if (useKnitr()) {
zeppelinR.setInterpreterOutput(null);
zeppelinR.set(".zcmd",
zeppelin git commit: ZEPPELIN-2515. After 100 minutes R process quits silently and spark.r interpreter becomes unresponsive
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 48e1c4bba -> 7a8400e96
ZEPPELIN-2515. After 100 minutes R process quits silently and spark.r
interpreter becomes unresponsive
### What is this PR for?
This PR would 2 features:
1. Make timeout of sparkr backend configurable.
2. Detect R backend dead and display proper message to frontend.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2515
### How should this be tested?
* unit test is added
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2839 from zjffdu/ZEPPELIN-2515 and squashes the following commits:
62888c5 [Jeff Zhang] ZEPPELIN-2515. After 100 minutes R process quits silently
and spark.r interpreter becomes unresponsive
(cherry picked from commit 8b2259cf5805f654b5da2725fa8207fa0b6d5f7e)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7a8400e9
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7a8400e9
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7a8400e9
Branch: refs/heads/branch-0.8
Commit: 7a8400e9616327e6be86278bd4b01b112a3ef031
Parents: 48e1c4b
Author: Jeff Zhang
Authored: Tue Mar 6 15:02:19 2018 +0800
Committer: Jeff Zhang
Committed: Wed Mar 14 20:54:47 2018 +0800
--
.../zeppelin/spark/SparkRInterpreter.java | 17 +--
.../org/apache/zeppelin/spark/ZeppelinR.java| 32 ++--
.../src/main/resources/R/zeppelin_sparkr.R | 3 +-
.../zeppelin/spark/SparkRInterpreterTest.java | 9 +-
4 files changed, 54 insertions(+), 7 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a8400e9/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
index 5efff0e..44f71b7 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
@@ -27,6 +27,7 @@ import org.apache.spark.SparkRBackend;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.zeppelin.interpreter.*;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
+import org.apache.zeppelin.interpreter.util.InterpreterOutputStream;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
import org.slf4j.Logger;
@@ -36,6 +37,8 @@ import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import java.util.Properties;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
/**
* R and SparkR interpreter with visualization support.
@@ -46,6 +49,7 @@ public class SparkRInterpreter extends Interpreter {
private static String renderOptions;
private SparkInterpreter sparkInterpreter;
private ZeppelinR zeppelinR;
+ private AtomicBoolean rbackendDead = new AtomicBoolean(false);
private SparkContext sc;
private JavaSparkContext jsc;
@@ -79,10 +83,11 @@ public class SparkRInterpreter extends Interpreter {
}
int port = SparkRBackend.port();
-
this.sparkInterpreter = getSparkInterpreter();
this.sc = sparkInterpreter.getSparkContext();
this.jsc = sparkInterpreter.getJavaSparkContext();
+int timeout = this.sc.getConf().getInt("spark.r.backendConnectionTimeout",
6000);
+
SparkVersion sparkVersion = new SparkVersion(sc.version());
ZeppelinRContext.setSparkContext(sc);
ZeppelinRContext.setJavaSparkContext(jsc);
@@ -92,7 +97,7 @@ public class SparkRInterpreter extends Interpreter {
ZeppelinRContext.setSqlContext(sparkInterpreter.getSQLContext());
ZeppelinRContext.setZeppelinContext(sparkInterpreter.getZeppelinContext());
-zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port, sparkVersion);
+zeppelinR = new ZeppelinR(rCmdPath, sparkRLibPath, port, sparkVersion,
timeout, this);
try {
zeppelinR.open();
} catch (IOException e) {
@@ -159,6 +164,10 @@ public class SparkRInterpreter extends Interpreter {
try {
// render output with knitr
+ if (rbackendDead.get()) {
+return new InterpreterResult(InterpreterResult.Code.ERROR,
+"sparkR backend is dead, please try to increase
spark.r.backendConnectionTimeout");
+
zeppelin git commit: [HOTFIX] Code style fix for branch-0.8
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 7a8400e96 -> c6485d621
[HOTFIX] Code style fix for branch-0.8
### What is this PR for?
HotFix for code style issue of branch-0.8
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* No jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2868 from zjffdu/HOTFIX_JavaDoc and squashes the following commits:
69f3fc9 [Jeff Zhang] [HOTFIX] Code sytle fix for branch-0.8
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c6485d62
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c6485d62
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c6485d62
Branch: refs/heads/branch-0.8
Commit: c6485d621a82356070f6b373c22b9d677b80c801
Parents: 7a8400e
Author: Jeff Zhang
Authored: Thu Mar 15 07:35:47 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 15 08:24:30 2018 +0800
--
.../main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java | 3 ++-
.../src/main/java/org/apache/zeppelin/spark/ZeppelinR.java| 3 +++
2 files changed, 5 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c6485d62/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
--
diff --git
a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
index 1eba9c2..f117686 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
@@ -668,7 +668,8 @@ public abstract class BaseLivyInterpreter extends
Interpreter {
} else {
restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
}
-restTemplate.getMessageConverters().add(0, new
StringHttpMessageConverter(Charset.forName("UTF-8")));
+restTemplate.getMessageConverters().add(0,
+new StringHttpMessageConverter(Charset.forName("UTF-8")));
return restTemplate;
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c6485d62/spark/interpreter/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
index e481dbe..edd3233 100644
--- a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
+++ b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/ZeppelinR.java
@@ -399,6 +399,9 @@ public class ZeppelinR implements ExecuteResultHandler {
}
+ /**
+ * InterpreterOutptStream for SparkInterpreter, used for checking R process
status
+ */
public static class SparkRInterpreterOutputStream extends
InterpreterOutputStream {
private SparkRInterpreter sparkRInterpreter;
zeppelin git commit: ZEPPELIN-3330. Add more test for RemoteInterpreterServer
Repository: zeppelin
Updated Branches:
refs/heads/master 6001b98de -> 5499fc4e9
ZEPPELIN-3330. Add more test for RemoteInterpreterServer
### What is this PR for?
This PR add more test for RemoteInterpreterServer for the purpose of further
changes on RemoveInterpreterServer. Besides, it did some code refactoring here.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3330
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2869 from zjffdu/ZEPPELIN-3330 and squashes the following commits:
8e8c219 [Jeff Zhang] ZEPPELIN-3330. Add more test for RemoteInterpreterServer
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/5499fc4e
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/5499fc4e
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/5499fc4e
Branch: refs/heads/master
Commit: 5499fc4e90b44269bd5e4c4a94f3dd95b6c6fedc
Parents: 6001b98
Author: Jeff Zhang
Authored: Wed Mar 14 16:05:50 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 15 13:23:23 2018 +0800
--
.../interpreter/BaseZeppelinContext.java| 6 +-
.../interpreter/InterpreterContext.java | 10 +-
.../remote/RemoteInterpreterServer.java | 29 +-
.../remote/RemoteInterpreterServerTest.java | 284 ++-
4 files changed, 234 insertions(+), 95 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5499fc4e/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
index e38a29f..2e9a9de 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
@@ -772,7 +772,7 @@ public abstract class BaseZeppelinContext {
*/
@Experimental
public void registerHook(String event, String cmd) {
-String className = interpreterContext.getClassName();
+String className = interpreterContext.getInterpreterClassName();
registerHook(event, cmd, className);
}
@@ -794,7 +794,7 @@ public abstract class BaseZeppelinContext {
*/
@Experimental
public String getHook(String event) {
-String className = interpreterContext.getClassName();
+String className = interpreterContext.getInterpreterClassName();
return getHook(event, className);
}
@@ -816,7 +816,7 @@ public abstract class BaseZeppelinContext {
*/
@Experimental
public void unregisterHook(String event) {
-String className = interpreterContext.getClassName();
+String className = interpreterContext.getInterpreterClassName();
unregisterHook(event, className);
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5499fc4e/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterContext.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterContext.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterContext.java
index 8fa0904..6157d69 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterContext.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/InterpreterContext.java
@@ -62,7 +62,7 @@ public class InterpreterContext {
private AngularObjectRegistry angularObjectRegistry;
private ResourcePool resourcePool;
private List runners = new ArrayList<>();
- private String className;
+ private String interpreterClassName;
private RemoteEventClientWrapper client;
private RemoteWorksController remoteWorksController;
private Map progressMap;
@@ -214,12 +214,12 @@ public class InterpreterContext {
return runners;
}
- public String getClassName() {
-return className;
+ public String getInterpreterClassName() {
+return interpreterClassName;
}
- public void setClassName(String className) {
-this.className = className;
+ public void setInterpreterClassName(String className) {
+this.interpreterClassName = className;
}
public RemoteEventClientWrapper getClient() {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5499fc4e/zeppelin-interp
zeppelin git commit: ZEPPELIN-3337. Add more test to SparkRInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master 5499fc4e9 -> e30fe73e9
ZEPPELIN-3337. Add more test to SparkRInterpreter
### What is this PR for?
Add more test for SparkRInterpreter, and also some code refactoring in
SparkRInterpreter. Also fix one bug of SparkRInterpreter that it can be
cancelled.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3337
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2871 from zjffdu/ZEPPELIN-3337 and squashes the following commits:
6cd91d5 [Jeff Zhang] ZEPPELIN-3337. Add more test to SparkRInterpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/e30fe73e
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/e30fe73e
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/e30fe73e
Branch: refs/heads/master
Commit: e30fe73e9a3bcd9cb14f02915883761894ceb2e4
Parents: 5499fc4
Author: Jeff Zhang
Authored: Thu Mar 15 12:38:01 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 15 22:23:55 2018 +0800
--
.../zeppelin/spark/SparkRInterpreter.java | 44 +--
.../zeppelin/spark/SparkRInterpreterTest.java | 59 +++-
testing/install_external_dependencies.sh| 1 +
.../interpreter/InterpreterContext.java | 11 +++-
.../interpreter/SparkIntegrationTest.java | 2 +-
5 files changed, 70 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e30fe73e/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
index 44f71b7..896f3a1 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
@@ -46,8 +46,9 @@ import java.util.concurrent.atomic.AtomicInteger;
public class SparkRInterpreter extends Interpreter {
private static final Logger logger =
LoggerFactory.getLogger(SparkRInterpreter.class);
- private static String renderOptions;
+ private String renderOptions;
private SparkInterpreter sparkInterpreter;
+ private boolean isSpark2;
private ZeppelinR zeppelinR;
private AtomicBoolean rbackendDead = new AtomicBoolean(false);
private SparkContext sc;
@@ -75,6 +76,7 @@ public class SparkRInterpreter extends Interpreter {
sparkRLibPath = "sparkr";
}
+// Share the same SparkRBackend across sessions
synchronized (SparkRBackend.backend()) {
if (!SparkRBackend.isStarted()) {
SparkRBackend.init();
@@ -86,12 +88,13 @@ public class SparkRInterpreter extends Interpreter {
this.sparkInterpreter = getSparkInterpreter();
this.sc = sparkInterpreter.getSparkContext();
this.jsc = sparkInterpreter.getJavaSparkContext();
+SparkVersion sparkVersion = new SparkVersion(sc.version());
+this.isSpark2 = sparkVersion.newerThanEquals(SparkVersion.SPARK_2_0_0);
int timeout = this.sc.getConf().getInt("spark.r.backendConnectionTimeout",
6000);
-SparkVersion sparkVersion = new SparkVersion(sc.version());
ZeppelinRContext.setSparkContext(sc);
ZeppelinRContext.setJavaSparkContext(jsc);
-if (Utils.isSpark2()) {
+if (isSpark2) {
ZeppelinRContext.setSparkSession(sparkInterpreter.getSparkSession());
}
ZeppelinRContext.setSqlContext(sparkInterpreter.getSQLContext());
@@ -101,37 +104,28 @@ public class SparkRInterpreter extends Interpreter {
try {
zeppelinR.open();
} catch (IOException e) {
- logger.error("Exception while opening SparkRInterpreter", e);
- throw new InterpreterException(e);
+ throw new InterpreterException("Exception while opening
SparkRInterpreter", e);
}
if (useKnitr()) {
zeppelinR.eval("library('knitr')");
}
-renderOptions = getProperty("zeppelin.R.render.options");
- }
-
- String getJobGroup(InterpreterContext context){
-return "zeppelin-" + context.getParagraphId();
+renderOptions = getProperty("zeppelin.R.render.options",
+"out.format = 'html', comment = NA, echo = FALSE, results = 'asis',
message = F, " +
+"warning = F, fig.retina = 2&
zeppelin git commit: ZEPPELIN-3337. Add more test to SparkRInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 257db1d5a -> e91837cd5
ZEPPELIN-3337. Add more test to SparkRInterpreter
### What is this PR for?
Add more test for SparkRInterpreter, and also some code refactoring in
SparkRInterpreter. Also fix one bug of SparkRInterpreter that it can be
cancelled.
### What type of PR is it?
[ Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3337
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2871 from zjffdu/ZEPPELIN-3337 and squashes the following commits:
6cd91d5 [Jeff Zhang] ZEPPELIN-3337. Add more test to SparkRInterpreter
(cherry picked from commit e30fe73e9a3bcd9cb14f02915883761894ceb2e4)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/e91837cd
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/e91837cd
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/e91837cd
Branch: refs/heads/branch-0.8
Commit: e91837cd5a22b47bcf5a75cda34067a1b1c2be6a
Parents: 257db1d
Author: Jeff Zhang
Authored: Thu Mar 15 12:38:01 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 15 22:26:59 2018 +0800
--
.../zeppelin/spark/SparkRInterpreter.java | 44 +--
.../zeppelin/spark/SparkRInterpreterTest.java | 59 +++-
testing/install_external_dependencies.sh| 1 +
.../interpreter/InterpreterContext.java | 11 +++-
.../interpreter/SparkIntegrationTest.java | 2 +-
5 files changed, 70 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/e91837cd/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
index 44f71b7..896f3a1 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/SparkRInterpreter.java
@@ -46,8 +46,9 @@ import java.util.concurrent.atomic.AtomicInteger;
public class SparkRInterpreter extends Interpreter {
private static final Logger logger =
LoggerFactory.getLogger(SparkRInterpreter.class);
- private static String renderOptions;
+ private String renderOptions;
private SparkInterpreter sparkInterpreter;
+ private boolean isSpark2;
private ZeppelinR zeppelinR;
private AtomicBoolean rbackendDead = new AtomicBoolean(false);
private SparkContext sc;
@@ -75,6 +76,7 @@ public class SparkRInterpreter extends Interpreter {
sparkRLibPath = "sparkr";
}
+// Share the same SparkRBackend across sessions
synchronized (SparkRBackend.backend()) {
if (!SparkRBackend.isStarted()) {
SparkRBackend.init();
@@ -86,12 +88,13 @@ public class SparkRInterpreter extends Interpreter {
this.sparkInterpreter = getSparkInterpreter();
this.sc = sparkInterpreter.getSparkContext();
this.jsc = sparkInterpreter.getJavaSparkContext();
+SparkVersion sparkVersion = new SparkVersion(sc.version());
+this.isSpark2 = sparkVersion.newerThanEquals(SparkVersion.SPARK_2_0_0);
int timeout = this.sc.getConf().getInt("spark.r.backendConnectionTimeout",
6000);
-SparkVersion sparkVersion = new SparkVersion(sc.version());
ZeppelinRContext.setSparkContext(sc);
ZeppelinRContext.setJavaSparkContext(jsc);
-if (Utils.isSpark2()) {
+if (isSpark2) {
ZeppelinRContext.setSparkSession(sparkInterpreter.getSparkSession());
}
ZeppelinRContext.setSqlContext(sparkInterpreter.getSQLContext());
@@ -101,37 +104,28 @@ public class SparkRInterpreter extends Interpreter {
try {
zeppelinR.open();
} catch (IOException e) {
- logger.error("Exception while opening SparkRInterpreter", e);
- throw new InterpreterException(e);
+ throw new InterpreterException("Exception while opening
SparkRInterpreter", e);
}
if (useKnitr()) {
zeppelinR.eval("library('knitr')");
}
-renderOptions = getProperty("zeppelin.R.render.options");
- }
-
- String getJobGroup(InterpreterContext context){
-return "zeppelin-" + context.getParagraphId();
+renderOptions = getProperty("zeppelin.R.render.options",
+"out.format = 'html', comment = NA, echo =
zeppelin git commit: ZEPPELIN-3339. Add more test for ZeppelinContext
Repository: zeppelin
Updated Branches:
refs/heads/master e30fe73e9 -> c6f7f2bdb
ZEPPELIN-3339. Add more test for ZeppelinContext
### What is this PR for?
Add more test for ZeppelinContext, especially for run api and ResourcePool
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3339
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2874 from zjffdu/ZEPPELIN-3339 and squashes the following commits:
83de188 [Jeff Zhang] ZEPPELIN-3339. Add more test for ZeppelinContext
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/c6f7f2bd
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/c6f7f2bd
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/c6f7f2bd
Branch: refs/heads/master
Commit: c6f7f2bdb3a41bb04508c800ec6ed4c411680b26
Parents: e30fe73
Author: Jeff Zhang
Authored: Thu Mar 15 16:04:57 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 16 14:46:33 2018 +0800
--
.../interpreter/BaseZeppelinContext.java| 4 +
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 360 ++-
2 files changed, 115 insertions(+), 249 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c6f7f2bd/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
index 2e9a9de..33d8626 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
@@ -28,6 +28,8 @@ import
org.apache.zeppelin.interpreter.remote.RemoteEventClientWrapper;
import org.apache.zeppelin.resource.Resource;
import org.apache.zeppelin.resource.ResourcePool;
import org.apache.zeppelin.resource.ResourceSet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Collection;
@@ -40,6 +42,7 @@ import java.util.Map;
*/
public abstract class BaseZeppelinContext {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(BaseZeppelinContext.class);
protected InterpreterContext interpreterContext;
protected int maxResult;
@@ -336,6 +339,7 @@ public abstract class BaseZeppelinContext {
if (r.getNoteId().equals(runningNoteId) &&
r.getParagraphId().equals(runningParagraphId)) {
continue;
}
+ LOGGER.debug("Run Paragraph: " + r.getParagraphId() + " of Note: " +
r.getNoteId());
r.run();
}
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/c6f7f2bd/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
--
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
index bda555a..14eb0d9 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
@@ -17,8 +17,7 @@
package org.apache.zeppelin.rest;
import org.apache.commons.io.FileUtils;
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterProperty;
import org.apache.zeppelin.interpreter.InterpreterResult;
@@ -37,19 +36,14 @@ import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
-import java.io.InputStream;
-import java.io.InputStreamReader;
-import java.net.URL;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertTrue;
/**
@@ -108,6 +102,7 @@ public class ZeppelinSparkClusterTest extends
AbstractTestRestApi {
@BeforeClass
public static void setUp() throws Excep
zeppelin git commit: ZEPPELIN-3339. Add more test for ZeppelinContext
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 e91837cd5 -> 45d07668f
ZEPPELIN-3339. Add more test for ZeppelinContext
### What is this PR for?
Add more test for ZeppelinContext, especially for run api and ResourcePool
### What type of PR is it?
[Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3339
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2874 from zjffdu/ZEPPELIN-3339 and squashes the following commits:
83de188 [Jeff Zhang] ZEPPELIN-3339. Add more test for ZeppelinContext
(cherry picked from commit c6f7f2bdb3a41bb04508c800ec6ed4c411680b26)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/45d07668
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/45d07668
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/45d07668
Branch: refs/heads/branch-0.8
Commit: 45d07668f95886f35d628db8caba9a48484d1eed
Parents: e91837c
Author: Jeff Zhang
Authored: Thu Mar 15 16:04:57 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 16 14:46:49 2018 +0800
--
.../interpreter/BaseZeppelinContext.java| 4 +
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 360 ++-
2 files changed, 115 insertions(+), 249 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/45d07668/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
--
diff --git
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
index e38a29f..3433c32 100644
---
a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
+++
b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/interpreter/BaseZeppelinContext.java
@@ -28,6 +28,8 @@ import
org.apache.zeppelin.interpreter.remote.RemoteEventClientWrapper;
import org.apache.zeppelin.resource.Resource;
import org.apache.zeppelin.resource.ResourcePool;
import org.apache.zeppelin.resource.ResourceSet;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.Collection;
@@ -40,6 +42,7 @@ import java.util.Map;
*/
public abstract class BaseZeppelinContext {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(BaseZeppelinContext.class);
protected InterpreterContext interpreterContext;
protected int maxResult;
@@ -336,6 +339,7 @@ public abstract class BaseZeppelinContext {
if (r.getNoteId().equals(runningNoteId) &&
r.getParagraphId().equals(runningParagraphId)) {
continue;
}
+ LOGGER.debug("Run Paragraph: " + r.getParagraphId() + " of Note: " +
r.getNoteId());
r.run();
}
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/45d07668/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
--
diff --git
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
index bda555a..14eb0d9 100644
---
a/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
+++
b/zeppelin-server/src/test/java/org/apache/zeppelin/rest/ZeppelinSparkClusterTest.java
@@ -17,8 +17,7 @@
package org.apache.zeppelin.rest;
import org.apache.commons.io.FileUtils;
-import org.apache.commons.io.IOUtils;
-import org.apache.commons.lang3.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterProperty;
import org.apache.zeppelin.interpreter.InterpreterResult;
@@ -37,19 +36,14 @@ import org.junit.runners.Parameterized;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import java.io.BufferedReader;
import java.io.File;
import java.io.IOException;
-import java.io.InputStream;
-import java.io.InputStreamReader;
-import java.net.URL;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotEquals;
import static org.junit.Assert.assertTrue;
/**
@@ -108,6 +102,7 @@ public class ZeppelinSparkClusterTest extends
Abstract
zeppelin git commit: ZEPPELIN-3343. Interpreter Hook is broken
Repository: zeppelin
Updated Branches:
refs/heads/master c6f7f2bdb -> 90b27cb67
ZEPPELIN-3343. Interpreter Hook is broken
### What is this PR for?
Currently the interpreter hook mechanism is broken. This PR fix this issue and
also add unit test for interpreter hook, (including global hook and note level
hook)
### What type of PR is it?
[Bug Fix | Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3343
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2875 from zjffdu/ZEPPELIN-3343 and squashes the following commits:
3c55f07 [Jeff Zhang] ZEPPELIN-3343. Interpreter Hook is broken
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/90b27cb6
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/90b27cb6
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/90b27cb6
Branch: refs/heads/master
Commit: 90b27cb678e94084a2dee70b616969424b5999c6
Parents: c6f7f2b
Author: Jeff Zhang
Authored: Fri Mar 16 16:05:31 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 16 17:51:50 2018 +0800
--
.../zeppelin/python/PythonInterpreter.java | 6 +-
.../resources/grpc/python/zeppelin_python.py| 23
.../main/resources/python/zeppelin_python.py| 24
.../zeppelin/spark/PySparkInterpreter.java | 7 +-
.../main/resources/python/zeppelin_pyspark.py | 12 ++
.../interpreter/BaseZeppelinContext.java| 85 +++-
.../zeppelin/interpreter/Interpreter.java | 4 +-
.../interpreter/InterpreterContext.java | 10 ++
.../interpreter/InterpreterHookRegistry.java| 70 +-
.../interpreter/InvalidHookException.java | 29
.../interpreter/LazyOpenInterpreter.java| 4 +-
.../remote/RemoteInterpreterServer.java | 21 ++-
.../interpreter/BaseZeppelinContextTest.java| 133 +++
.../InterpreterHookRegistryTest.java| 54
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 31 -
15 files changed, 403 insertions(+), 110 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/90b27cb6/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
index 1864409..f646de3 100644
--- a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -259,7 +259,11 @@ public class PythonInterpreter extends Interpreter
implements ExecuteResultHandl
// Add matplotlib display hook
InterpreterGroup intpGroup = getInterpreterGroup();
if (intpGroup != null && intpGroup.getInterpreterHookRegistry() != null) {
- registerHook(HookType.POST_EXEC_DEV, "__zeppelin__._displayhook()");
+ try {
+registerHook(HookType.POST_EXEC_DEV.getName(),
"__zeppelin__._displayhook()");
+ } catch (InvalidHookException e) {
+throw new InterpreterException(e);
+ }
}
// Add matplotlib display hook
try {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/90b27cb6/python/src/main/resources/grpc/python/zeppelin_python.py
--
diff --git a/python/src/main/resources/grpc/python/zeppelin_python.py
b/python/src/main/resources/grpc/python/zeppelin_python.py
index 6d7c33e..d76bdf4 100644
--- a/python/src/main/resources/grpc/python/zeppelin_python.py
+++ b/python/src/main/resources/grpc/python/zeppelin_python.py
@@ -113,6 +113,29 @@ class PyZeppelinContext(object):
#)
body_buf.close(); header_buf.close()
+ def registerHook(self, event, cmd, replName=None):
+if replName is None:
+ self.z.registerHook(event, cmd)
+else:
+ self.z.registerHook(event, cmd, replName)
+
+ def unregisterHook(self, event, replName=None):
+if replName is None:
+ self.z.unregisterHook(event)
+else:
+ self.z.unregisterHook(event, replName)
+
+ def registerNoteHook(self, event, cmd, noteId, replName=None):
+if replName is None:
+ self.z.registerNoteHook(event, cmd, noteId)
+else:
+ self.z.registerNoteHook(event, cmd, noteId, replName)
+
+ def unregisterNoteHook(self, event, noteId, replName=None):
+if replName is None:
+ self.z.unregisterNoteHook(event, noteId)
+else:
+ self.z.unregisterN
zeppelin git commit: ZEPPELIN-3343. Interpreter Hook is broken
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 45d07668f -> 27fee5469
ZEPPELIN-3343. Interpreter Hook is broken
Currently the interpreter hook mechanism is broken. This PR fix this issue and
also add unit test for interpreter hook, (including global hook and note level
hook)
[Bug Fix | Improvement | Refactoring]
* [ ] - Task
* https://issues.apache.org/jira/browse/ZEPPELIN-3343
* CI pass
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2875 from zjffdu/ZEPPELIN-3343 and squashes the following commits:
3c55f07 [Jeff Zhang] ZEPPELIN-3343. Interpreter Hook is broken
(cherry picked from commit 90b27cb678e94084a2dee70b616969424b5999c6)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/27fee546
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/27fee546
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/27fee546
Branch: refs/heads/branch-0.8
Commit: 27fee54692f6e4664f80155672943b60d5ef1648
Parents: 45d0766
Author: Jeff Zhang
Authored: Fri Mar 16 16:05:31 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 16 17:57:51 2018 +0800
--
.../zeppelin/python/PythonInterpreter.java | 6 +-
.../resources/grpc/python/zeppelin_python.py| 23
.../main/resources/python/zeppelin_python.py| 24
.../zeppelin/spark/PySparkInterpreter.java | 7 +-
.../main/resources/python/zeppelin_pyspark.py | 12 ++
.../interpreter/BaseZeppelinContext.java| 85 +++-
.../zeppelin/interpreter/Interpreter.java | 4 +-
.../interpreter/InterpreterContext.java | 10 ++
.../interpreter/InterpreterHookRegistry.java| 70 +-
.../interpreter/InvalidHookException.java | 29
.../interpreter/LazyOpenInterpreter.java| 4 +-
.../remote/RemoteInterpreterServer.java | 21 ++-
.../interpreter/BaseZeppelinContextTest.java| 133 +++
.../InterpreterHookRegistryTest.java| 72 +-
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 31 -
15 files changed, 411 insertions(+), 120 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/27fee546/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
index 1864409..f646de3 100644
--- a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -259,7 +259,11 @@ public class PythonInterpreter extends Interpreter
implements ExecuteResultHandl
// Add matplotlib display hook
InterpreterGroup intpGroup = getInterpreterGroup();
if (intpGroup != null && intpGroup.getInterpreterHookRegistry() != null) {
- registerHook(HookType.POST_EXEC_DEV, "__zeppelin__._displayhook()");
+ try {
+registerHook(HookType.POST_EXEC_DEV.getName(),
"__zeppelin__._displayhook()");
+ } catch (InvalidHookException e) {
+throw new InterpreterException(e);
+ }
}
// Add matplotlib display hook
try {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/27fee546/python/src/main/resources/grpc/python/zeppelin_python.py
--
diff --git a/python/src/main/resources/grpc/python/zeppelin_python.py
b/python/src/main/resources/grpc/python/zeppelin_python.py
index 6d7c33e..d76bdf4 100644
--- a/python/src/main/resources/grpc/python/zeppelin_python.py
+++ b/python/src/main/resources/grpc/python/zeppelin_python.py
@@ -113,6 +113,29 @@ class PyZeppelinContext(object):
#)
body_buf.close(); header_buf.close()
+ def registerHook(self, event, cmd, replName=None):
+if replName is None:
+ self.z.registerHook(event, cmd)
+else:
+ self.z.registerHook(event, cmd, replName)
+
+ def unregisterHook(self, event, replName=None):
+if replName is None:
+ self.z.unregisterHook(event)
+else:
+ self.z.unregisterHook(event, replName)
+
+ def registerNoteHook(self, event, cmd, noteId, replName=None):
+if replName is None:
+ self.z.registerNoteHook(event, cmd, noteId)
+else:
+ self.z.registerNoteHook(event, cmd, noteId, replName)
+
+ def unregisterNoteHook(self, event, noteId, replName=None):
+if replName is None:
+ self.z.unregisterNoteHook(event, noteId)
+else:
+ self.z.unregisterNoteHook(event, noteId, replName)
# start JVM gateway
client = G
zeppelin git commit: ZEPPELIN-3345. Don't load interpreter setting when it exist interpreter.json but no interpreter-setting.json found
Repository: zeppelin
Updated Branches:
refs/heads/master 90b27cb67 -> a63f04147
ZEPPELIN-3345. Don't load interpreter setting when it exist interpreter.json
but no interpreter-setting.json found
### What is this PR for?
It usually happens in dev enviroment. e.g. I have livy interpreter in
interpreter.json due to history testing but no livy folder under
`{ZEPPELIN_HOME}/interpreter` after I rebuilt the project without building livy
module. This PR would just remove that invalid interpreter setting when loading
from interpreter.json
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3345
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2877 from zjffdu/ZEPPELIN-3345 and squashes the following commits:
bf72ead [Jeff Zhang] ZEPPELIN-3345. Don't load interpreter setting when it
exist interpreter.json but no interpreter-setting.json found
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a63f0414
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a63f0414
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a63f0414
Branch: refs/heads/master
Commit: a63f041477737c67177de292c90109f4b8ce2b1c
Parents: 90b27cb
Author: Jeff Zhang
Authored: Sat Mar 17 11:34:14 2018 +0800
Committer: Jeff Zhang
Committed: Sat Mar 17 13:01:30 2018 +0800
--
.../interpreter/InterpreterSettingManager.java| 18 --
1 file changed, 16 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a63f0414/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 9587a88..23d086d 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -215,7 +215,9 @@ public class InterpreterSettingManager {
List oldSettingIdList = entry.getValue();
List newSettingIdList = new ArrayList<>();
for (String oldId : oldSettingIdList) {
-
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+if (infoSaving.interpreterSettings.containsKey(oldId)) {
+
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+};
}
newBindingMap.put(noteId, newSettingIdList);
}
@@ -256,7 +258,19 @@ public class InterpreterSettingManager {
interpreterSettingTemplate.getInterpreterRunner());
} else {
LOGGER.warn("No InterpreterSetting Template found for
InterpreterSetting: "
-+ savedInterpreterSetting.getGroup());
++ savedInterpreterSetting.getGroup() + ", but it is found in
interpreter.json, "
++ "it would be skipped.");
+// also delete its binding
+for (Map.Entry> entry :
interpreterBindings.entrySet()) {
+ List ids = entry.getValue();
+ Iterator iter = ids.iterator();
+ while(iter.hasNext()) {
+if (iter.next().equals(savedInterpreterSetting.getId())) {
+ iter.remove();
+}
+ }
+}
+continue;
}
// Overwrite the default InterpreterSetting we registered from
InterpreterSetting Templates
zeppelin git commit: ZEPPELIN-3331. Use NullLifecycleManager by default
Repository: zeppelin Updated Branches: refs/heads/master a63f04147 -> 233c09873 ZEPPELIN-3331. Use NullLifecycleManager by default ### What is this PR for? Just keep the behavior consistent with previous version. So use NullLifecycleManager by default, user need to configure zeppelin-site.xml to enable TimeoutLifecycleManager ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3331 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2878 from zjffdu/ZEPPELIN-3331 and squashes the following commits: 5c94e36 [Jeff Zhang] ZEPPELIN-3331. Use NullLifecycleManager by default Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/233c0987 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/233c0987 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/233c0987 Branch: refs/heads/master Commit: 233c09873456ac0893ce30970b7caa64204d99ca Parents: a63f041 Author: Jeff Zhang Authored: Wed Mar 14 21:09:54 2018 +0800 Committer: Jeff Zhang Committed: Sat Mar 17 19:12:23 2018 +0800 -- conf/zeppelin-site.xml.template| 2 ++ .../main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java | 2 +- .../apache/zeppelin/interpreter/InterpreterSettingManagerTest.java | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/233c0987/conf/zeppelin-site.xml.template -- diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template index d39b19c..b12a072 100755 --- a/conf/zeppelin-site.xml.template +++ b/conf/zeppelin-site.xml.template @@ -434,6 +434,7 @@ Enable directory listings on server. +
zeppelin git commit: ZEPPELIN-3331. Use NullLifecycleManager by default
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 27fee5469 -> b08bf5ebd ZEPPELIN-3331. Use NullLifecycleManager by default ### What is this PR for? Just keep the behavior consistent with previous version. So use NullLifecycleManager by default, user need to configure zeppelin-site.xml to enable TimeoutLifecycleManager ### What type of PR is it? [ Improvement ] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3331 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2878 from zjffdu/ZEPPELIN-3331 and squashes the following commits: 5c94e36 [Jeff Zhang] ZEPPELIN-3331. Use NullLifecycleManager by default (cherry picked from commit 233c09873456ac0893ce30970b7caa64204d99ca) Signed-off-by: Jeff Zhang Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b08bf5eb Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b08bf5eb Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b08bf5eb Branch: refs/heads/branch-0.8 Commit: b08bf5ebd14a1d125eb1c64e14d74b84bff24fec Parents: 27fee54 Author: Jeff Zhang Authored: Wed Mar 14 21:09:54 2018 +0800 Committer: Jeff Zhang Committed: Sat Mar 17 19:12:39 2018 +0800 -- conf/zeppelin-site.xml.template| 2 ++ .../main/java/org/apache/zeppelin/conf/ZeppelinConfiguration.java | 2 +- .../apache/zeppelin/interpreter/InterpreterSettingManagerTest.java | 2 ++ 3 files changed, 5 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b08bf5eb/conf/zeppelin-site.xml.template -- diff --git a/conf/zeppelin-site.xml.template b/conf/zeppelin-site.xml.template index d39b19c..b12a072 100755 --- a/conf/zeppelin-site.xml.template +++ b/conf/zeppelin-site.xml.template @@ -434,6 +434,7 @@ Enable directory listings on server. +
zeppelin git commit: ZEPPELIN-3345. Don't load interpreter setting when it exist interpreter.json but no interpreter-setting.json found
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 b08bf5ebd -> cc2894c38
ZEPPELIN-3345. Don't load interpreter setting when it exist interpreter.json
but no interpreter-setting.json found
### What is this PR for?
It usually happens in dev enviroment. e.g. I have livy interpreter in
interpreter.json due to history testing but no livy folder under
`{ZEPPELIN_HOME}/interpreter` after I rebuilt the project without building livy
module. This PR would just remove that invalid interpreter setting when loading
from interpreter.json
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3345
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2877 from zjffdu/ZEPPELIN-3345 and squashes the following commits:
bf72ead [Jeff Zhang] ZEPPELIN-3345. Don't load interpreter setting when it
exist interpreter.json but no interpreter-setting.json found
(cherry picked from commit a63f041477737c67177de292c90109f4b8ce2b1c)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/cc2894c3
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/cc2894c3
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/cc2894c3
Branch: refs/heads/branch-0.8
Commit: cc2894c38ed8eaf5c0fb9c1da04fedcc588fb564
Parents: b08bf5e
Author: Jeff Zhang
Authored: Sat Mar 17 11:34:14 2018 +0800
Committer: Jeff Zhang
Committed: Sat Mar 17 21:30:40 2018 +0800
--
.../interpreter/InterpreterSettingManager.java| 18 --
1 file changed, 16 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/cc2894c3/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 9587a88..23d086d 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -215,7 +215,9 @@ public class InterpreterSettingManager {
List oldSettingIdList = entry.getValue();
List newSettingIdList = new ArrayList<>();
for (String oldId : oldSettingIdList) {
-
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+if (infoSaving.interpreterSettings.containsKey(oldId)) {
+
newSettingIdList.add(infoSaving.interpreterSettings.get(oldId).getName());
+};
}
newBindingMap.put(noteId, newSettingIdList);
}
@@ -256,7 +258,19 @@ public class InterpreterSettingManager {
interpreterSettingTemplate.getInterpreterRunner());
} else {
LOGGER.warn("No InterpreterSetting Template found for
InterpreterSetting: "
-+ savedInterpreterSetting.getGroup());
++ savedInterpreterSetting.getGroup() + ", but it is found in
interpreter.json, "
++ "it would be skipped.");
+// also delete its binding
+for (Map.Entry> entry :
interpreterBindings.entrySet()) {
+ List ids = entry.getValue();
+ Iterator iter = ids.iterator();
+ while(iter.hasNext()) {
+if (iter.next().equals(savedInterpreterSetting.getId())) {
+ iter.remove();
+}
+ }
+}
+continue;
}
// Overwrite the default InterpreterSetting we registered from
InterpreterSetting Templates
zeppelin git commit: ZEPPELIN-3208. Use interpreter setting's name as its id
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 cc2894c38 -> 97e6ef74b
ZEPPELIN-3208. Use interpreter setting's name as its id
### What is this PR for?
Interpreter setting's name is unique, so it is not necessary to use id to
identify it. For now we use a random string to represent such id which is not
easy to read especially when you read log for diagnose. But for backward
compatibility, I will keep id and set it as the same as interpreter setting's
name
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3208
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2778 from zjffdu/ZEPPELIN-3208 and squashes the following commits:
56a33c2 [Jeff Zhang] ZEPPELIN-3208. Use interpreter setting's name as its id
(cherry picked from commit b3ba458bd22f4320162cbb4baf3bab48a6844b30)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/97e6ef74
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/97e6ef74
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/97e6ef74
Branch: refs/heads/branch-0.8
Commit: 97e6ef74b07ef9204d3e4d7023e14d603b9d33d3
Parents: cc2894c
Author: Jeff Zhang
Authored: Wed Feb 7 18:25:46 2018 +0800
Committer: Jeff Zhang
Committed: Mon Mar 19 09:48:28 2018 +0800
--
.../java/org/apache/zeppelin/interpreter/InterpreterSetting.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/97e6ef74/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index b63c98a..a3b7f8d 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -261,6 +261,7 @@ public class InterpreterSetting {
void postProcessing() {
this.status = Status.READY;
+this.id = this.name;
if (this.lifecycleManager == null) {
this.lifecycleManager = new NullLifecycleManager(conf);
}
@@ -280,7 +281,7 @@ public class InterpreterSetting {
*/
public InterpreterSetting(InterpreterSetting o) {
this();
-this.id = generateId();
+this.id = o.name;
this.name = o.name;
this.group = o.group;
this.properties = convertInterpreterProperties(
zeppelin git commit: [HOTFIX] Codestyle fix for branch-0.8
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 97e6ef74b -> 4aacb55d8
[HOTFIX] Codestyle fix for branch-0.8
### What is this PR for?
trivial fix for branch-0.8 code style
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* No jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2879 from zjffdu/HOTFIX_codestyle and squashes the following commits:
16057b9 [Jeff Zhang] [HOTFIX] code sytle fix for branch-0.8
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/4aacb55d
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/4aacb55d
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/4aacb55d
Branch: refs/heads/branch-0.8
Commit: 4aacb55d823993a2c41e5692e24576e01fb37d20
Parents: 97e6ef7
Author: Jeff Zhang
Authored: Sat Mar 17 21:38:57 2018 +0800
Committer: Jeff Zhang
Committed: Mon Mar 19 09:53:00 2018 +0800
--
.../org/apache/zeppelin/interpreter/InterpreterSettingManager.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4aacb55d/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
index 23d086d..5eacbbb 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSettingManager.java
@@ -264,7 +264,7 @@ public class InterpreterSettingManager {
for (Map.Entry> entry :
interpreterBindings.entrySet()) {
List ids = entry.getValue();
Iterator iter = ids.iterator();
- while(iter.hasNext()) {
+ while (iter.hasNext()) {
if (iter.next().equals(savedInterpreterSetting.getId())) {
iter.remove();
}
zeppelin git commit: ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for PySparkInterpreter'
Repository: zeppelin
Updated Branches:
refs/heads/master 2a5960bd5 -> 67582b87e
ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for
PySparkInterpreter'
### What is this PR for?
Trivial PR for removing the unnecessary message
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3290
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2882 from zjffdu/ZEPPELIN-3290 and squashes the following commits:
7bf8d6e [Jeff Zhang] ZEPPELIN-3290. Unnecessary message 'IPython is available,
use IPython for PySparkInterpreter'
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/67582b87
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/67582b87
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/67582b87
Branch: refs/heads/master
Commit: 67582b87ecd61f661aaebdc66aea817ff2fd2c7f
Parents: 2a5960b
Author: Jeff Zhang
Authored: Tue Mar 20 09:24:20 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 20 14:09:14 2018 +0800
--
.../zeppelin/python/PythonInterpreter.java | 23 +---
.../zeppelin/spark/PySparkInterpreter.java | 23 +---
2 files changed, 10 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/67582b87/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
index f646de3..fab2ed9 100644
--- a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -232,30 +232,17 @@ public class PythonInterpreter extends Interpreter
implements ExecuteResultHandl
StringUtils.isEmpty(iPythonInterpreter.checkIPythonPrerequisite(getPythonBindPath(
{
try {
iPythonInterpreter.open();
-if (InterpreterContext.get() != null) {
- InterpreterContext.get().out.write(("IPython is available, " +
- "use IPython for PythonInterpreter\n")
- .getBytes());
-}
-LOG.info("Use IPythonInterpreter to replace PythonInterpreter");
+LOG.info("IPython is available, Use IPythonInterpreter to replace
PythonInterpreter");
return;
} catch (Exception e) {
iPythonInterpreter = null;
+LOG.warn("Fail to open IPythonInterpreter", e);
}
}
-// reset iPythonInterpreter to null
-iPythonInterpreter = null;
-
-try {
- if (InterpreterContext.get() != null) {
-InterpreterContext.get().out.write(("IPython is not available, " +
-"use the native PythonInterpreter\n")
-.getBytes());
- }
-} catch (IOException e) {
- LOG.warn("Fail to write InterpreterOutput", e.getMessage());
-}
+// reset iPythonInterpreter to null as it is not available
+iPythonInterpreter = null;
+LOG.info("IPython is not available, use the native PythonInterpreter");
// Add matplotlib display hook
InterpreterGroup intpGroup = getInterpreterGroup();
if (intpGroup != null && intpGroup.getInterpreterHookRegistry() != null) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/67582b87/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index bd2a30c..a838667 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -126,30 +126,17 @@ public class PySparkInterpreter extends Interpreter
implements ExecuteResultHand
iPySparkInterpreter.checkIPythonPrerequisite(getPythonExec(getProperties() {
try {
iPySparkInterpreter.open();
-if (InterpreterContext.get() != null) {
- // don't print it when it is in testing, just for easy output check
in test.
- InterpreterContext.get().out.write(("IPython is available, " +
-
zeppelin git commit: ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for PySparkInterpreter'
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 714eb963d -> 4e2216477
ZEPPELIN-3290. Unnecessary message 'IPython is available, use IPython for
PySparkInterpreter'
### What is this PR for?
Trivial PR for removing the unnecessary message
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3290
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2882 from zjffdu/ZEPPELIN-3290 and squashes the following commits:
7bf8d6e [Jeff Zhang] ZEPPELIN-3290. Unnecessary message 'IPython is available,
use IPython for PySparkInterpreter'
(cherry picked from commit 67582b87ecd61f661aaebdc66aea817ff2fd2c7f)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/4e221647
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/4e221647
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/4e221647
Branch: refs/heads/branch-0.8
Commit: 4e2216477a6f9787d3b0cd57fef9debdbfd26d16
Parents: 714eb96
Author: Jeff Zhang
Authored: Tue Mar 20 09:24:20 2018 +0800
Committer: Jeff Zhang
Committed: Tue Mar 20 14:09:28 2018 +0800
--
.../zeppelin/python/PythonInterpreter.java | 23 +---
.../zeppelin/spark/PySparkInterpreter.java | 23 +---
2 files changed, 10 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4e221647/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
index f646de3..fab2ed9 100644
--- a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -232,30 +232,17 @@ public class PythonInterpreter extends Interpreter
implements ExecuteResultHandl
StringUtils.isEmpty(iPythonInterpreter.checkIPythonPrerequisite(getPythonBindPath(
{
try {
iPythonInterpreter.open();
-if (InterpreterContext.get() != null) {
- InterpreterContext.get().out.write(("IPython is available, " +
- "use IPython for PythonInterpreter\n")
- .getBytes());
-}
-LOG.info("Use IPythonInterpreter to replace PythonInterpreter");
+LOG.info("IPython is available, Use IPythonInterpreter to replace
PythonInterpreter");
return;
} catch (Exception e) {
iPythonInterpreter = null;
+LOG.warn("Fail to open IPythonInterpreter", e);
}
}
-// reset iPythonInterpreter to null
-iPythonInterpreter = null;
-
-try {
- if (InterpreterContext.get() != null) {
-InterpreterContext.get().out.write(("IPython is not available, " +
-"use the native PythonInterpreter\n")
-.getBytes());
- }
-} catch (IOException e) {
- LOG.warn("Fail to write InterpreterOutput", e.getMessage());
-}
+// reset iPythonInterpreter to null as it is not available
+iPythonInterpreter = null;
+LOG.info("IPython is not available, use the native PythonInterpreter");
// Add matplotlib display hook
InterpreterGroup intpGroup = getInterpreterGroup();
if (intpGroup != null && intpGroup.getInterpreterHookRegistry() != null) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/4e221647/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index bd2a30c..a838667 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -126,30 +126,17 @@ public class PySparkInterpreter extends Interpreter
implements ExecuteResultHand
iPySparkInterpreter.checkIPythonPrerequisite(getPythonExec(getProperties() {
try {
iPySparkInterpreter.open();
-if (InterpreterContext.get() != null) {
- // don't print it when it is in testing, just for easy output check
in test.
zeppelin git commit: ZEPPELIN-3357. Livy security mode is broken
Repository: zeppelin
Updated Branches:
refs/heads/master 67582b87e -> db716c8b1
ZEPPELIN-3357. Livy security mode is broken
### What is this PR for?
Straightforward fix for livy security issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3357
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2887 from zjffdu/ZEPPELIN-3357 and squashes the following commits:
2fbbbeb [Jeff Zhang] ZEPPELIN-3357. Livy security mode is broken
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/db716c8b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/db716c8b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/db716c8b
Branch: refs/heads/master
Commit: db716c8b155428204dc12123bf2841c6e88b792b
Parents: 67582b8
Author: Jeff Zhang
Authored: Thu Mar 22 15:51:52 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 22 15:54:03 2018 +0800
--
.../java/org/apache/zeppelin/livy/BaseLivyInterpreter.java | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/db716c8b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
--
diff --git
a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
index 1eba9c2..de0fff2 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
@@ -662,11 +662,12 @@ public abstract class BaseLivyInterpreter extends
Interpreter {
} else {
restTemplate = new KerberosRestTemplate(keytabLocation, principal,
httpClient);
}
-}
-if (httpClient == null) {
- restTemplate = new RestTemplate();
} else {
- restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ if (httpClient == null) {
+restTemplate = new RestTemplate();
+ } else {
+restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ }
}
restTemplate.getMessageConverters().add(0, new
StringHttpMessageConverter(Charset.forName("UTF-8")));
return restTemplate;
zeppelin git commit: ZEPPELIN-3357. Livy security mode is broken
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 4e2216477 -> f5e56f1ee
ZEPPELIN-3357. Livy security mode is broken
### What is this PR for?
Straightforward fix for livy security issue.
### What type of PR is it?
[Bug Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3357
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2887 from zjffdu/ZEPPELIN-3357 and squashes the following commits:
2fbbbeb [Jeff Zhang] ZEPPELIN-3357. Livy security mode is broken
(cherry picked from commit db716c8b155428204dc12123bf2841c6e88b792b)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/f5e56f1e
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/f5e56f1e
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/f5e56f1e
Branch: refs/heads/branch-0.8
Commit: f5e56f1ee45c536a5f5569a6f23a9d06a3667ae7
Parents: 4e22164
Author: Jeff Zhang
Authored: Thu Mar 22 15:51:52 2018 +0800
Committer: Jeff Zhang
Committed: Thu Mar 22 15:54:21 2018 +0800
--
.../java/org/apache/zeppelin/livy/BaseLivyInterpreter.java | 9 +
1 file changed, 5 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/f5e56f1e/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
--
diff --git
a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
index f117686..858145c 100644
--- a/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
+++ b/livy/src/main/java/org/apache/zeppelin/livy/BaseLivyInterpreter.java
@@ -662,11 +662,12 @@ public abstract class BaseLivyInterpreter extends
Interpreter {
} else {
restTemplate = new KerberosRestTemplate(keytabLocation, principal,
httpClient);
}
-}
-if (httpClient == null) {
- restTemplate = new RestTemplate();
} else {
- restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ if (httpClient == null) {
+restTemplate = new RestTemplate();
+ } else {
+restTemplate = new RestTemplate(new
HttpComponentsClientHttpRequestFactory(httpClient));
+ }
}
restTemplate.getMessageConverters().add(0,
new StringHttpMessageConverter(Charset.forName("UTF-8")));
[4/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-plugins/notebookrepo/zeppelin-hub/src/main/java/org/apache/zeppelin/notebook/repo/zeppelinhub/ZeppelinHubRepo.java
--
diff --git
a/zeppelin-plugins/notebookrepo/zeppelin-hub/src/main/java/org/apache/zeppelin/notebook/repo/zeppelinhub/ZeppelinHubRepo.java
b/zeppelin-plugins/notebookrepo/zeppelin-hub/src/main/java/org/apache/zeppelin/notebook/repo/zeppelinhub/ZeppelinHubRepo.java
new file mode 100644
index 000..9dd9fbf
--- /dev/null
+++
b/zeppelin-plugins/notebookrepo/zeppelin-hub/src/main/java/org/apache/zeppelin/notebook/repo/zeppelinhub/ZeppelinHubRepo.java
@@ -0,0 +1,385 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.zeppelin.notebook.repo.zeppelinhub;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.notebook.Note;
+import org.apache.zeppelin.notebook.NoteInfo;
+import org.apache.zeppelin.notebook.repo.NotebookRepoWithVersionControl;
+import org.apache.zeppelin.notebook.repo.NotebookRepoSettingsInfo;
+import org.apache.zeppelin.notebook.repo.zeppelinhub.model.Instance;
+import org.apache.zeppelin.notebook.repo.zeppelinhub.model.UserTokenContainer;
+import
org.apache.zeppelin.notebook.repo.zeppelinhub.model.UserSessionContainer;
+import
org.apache.zeppelin.notebook.repo.zeppelinhub.rest.ZeppelinhubRestApiHandler;
+import org.apache.zeppelin.notebook.repo.zeppelinhub.websocket.Client;
+import
org.apache.zeppelin.notebook.repo.zeppelinhub.websocket.utils.ZeppelinhubUtils;
+import org.apache.zeppelin.user.AuthenticationInfo;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Joiner;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.gson.Gson;
+import com.google.gson.reflect.TypeToken;
+
+/**
+ * ZeppelinHub repo class.
+ */
+public class ZeppelinHubRepo implements NotebookRepoWithVersionControl {
+ private static final Logger LOG =
LoggerFactory.getLogger(ZeppelinHubRepo.class);
+ private static final String DEFAULT_SERVER = "https://www.zeppelinhub.com";;
+ static final String ZEPPELIN_CONF_PROP_NAME_SERVER =
"zeppelinhub.api.address";
+ static final String ZEPPELIN_CONF_PROP_NAME_TOKEN = "zeppelinhub.api.token";
+
+ private static final Gson GSON = new Gson();
+ private static final Note EMPTY_NOTE = new Note();
+ private Client websocketClient;
+ private UserTokenContainer tokenManager;
+
+ private String token;
+ private ZeppelinhubRestApiHandler restApiClient;
+
+ private ZeppelinConfiguration conf;
+
+ public ZeppelinHubRepo() {
+
+ }
+
+ public ZeppelinHubRepo(ZeppelinConfiguration conf) {
+this();
+init(conf);
+ }
+
+ public void init(ZeppelinConfiguration conf) {
+this.conf = conf;
+String zeppelinHubUrl = getZeppelinHubUrl(conf);
+LOG.info("Initializing ZeppelinHub integration module");
+
+token = conf.getString("ZEPPELINHUB_API_TOKEN",
ZEPPELIN_CONF_PROP_NAME_TOKEN, "");
+restApiClient = ZeppelinhubRestApiHandler.newInstance(zeppelinHubUrl);
+//TODO(khalid): check which realm for authentication, pass to token manager
+tokenManager = UserTokenContainer.init(restApiClient, token);
+
+websocketClient = Client.initialize(getZeppelinWebsocketUri(conf),
+getZeppelinhubWebsocketUri(conf), token, conf);
+websocketClient.start();
+ }
+
+ private String getZeppelinHubWsUri(URI api) throws URISyntaxException {
+URI apiRoot = api;
+String scheme = apiRoot.getScheme();
+int port = apiRoot.getPort();
+if (port <= 0) {
+ port = (scheme != null && scheme.equals("https")) ? 443 : 80;
+}
+
+if (scheme == null) {
+ LOG.info("{} is not a valid zeppelinhub server address. proceed with
default address {}",
+ apiRoot, DEFAULT_SERVER);
+ apiRoot = new URI(DEFAULT_SERVER);
+ scheme = apiRoot.getScheme();
+ port = apiRoot.ge
[6/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-plugins/notebookrepo/git/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java
--
diff --git
a/zeppelin-plugins/notebookrepo/git/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java
b/zeppelin-plugins/notebookrepo/git/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java
new file mode 100644
index 000..7729d52
--- /dev/null
+++
b/zeppelin-plugins/notebookrepo/git/src/main/java/org/apache/zeppelin/notebook/repo/GitNotebookRepo.java
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.notebook.repo;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Joiner;
+import com.google.common.collect.Lists;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.notebook.Note;
+import org.apache.zeppelin.user.AuthenticationInfo;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.api.errors.NoHeadException;
+import org.eclipse.jgit.diff.DiffEntry;
+import org.eclipse.jgit.dircache.DirCache;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.Constants;
+import org.eclipse.jgit.lib.ObjectId;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.eclipse.jgit.treewalk.filter.PathFilter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Collection;
+import java.util.List;
+
+/**
+ * NotebookRepo that hosts all the notebook FS in a single Git repo
+ *
+ * This impl intended to be simple and straightforward:
+ * - does not handle branches
+ * - only basic local git file repo, no remote Github push\pull. GitHub
integration is
+ * implemented in @see {@link
org.apache.zeppelin.notebook.repo.GitNotebookRepo}
+ *
+ * TODO(bzz): add default .gitignore
+ */
+public class GitNotebookRepo extends VFSNotebookRepo implements
NotebookRepoWithVersionControl {
+ private static final Logger LOG =
LoggerFactory.getLogger(GitNotebookRepo.class);
+
+ private String localPath;
+ private Git git;
+
+ public GitNotebookRepo() {
+super();
+ }
+
+ @VisibleForTesting
+ public GitNotebookRepo(ZeppelinConfiguration conf) throws IOException {
+this();
+init(conf);
+ }
+
+ @Override
+ public void init(ZeppelinConfiguration conf) throws IOException {
+//TODO(zjffdu), it is weird that I can not call super.init directly here,
as it would cause
+//AbstractMethodError
+this.conf = conf;
+setNotebookDirectory(conf.getNotebookDir());
+
+localPath = getRootDir().getName().getPath();
+LOG.info("Opening a git repo at '{}'", localPath);
+Repository localRepo = new
FileRepository(Joiner.on(File.separator).join(localPath, ".git"));
+if (!localRepo.getDirectory().exists()) {
+ LOG.info("Git repo {} does not exist, creating a new one",
localRepo.getDirectory());
+ localRepo.create();
+}
+git = new Git(localRepo);
+ }
+
+ @Override
+ public synchronized void save(Note note, AuthenticationInfo subject) throws
IOException {
+super.save(note, subject);
+ }
+
+ /* implemented as git add+commit
+ * @param pattern is the noteId
+ * @param commitMessage is a commit message (checkpoint message)
+ * (non-Javadoc)
+ * @see org.apache.zeppelin.notebook.repo.VFSNotebookRepo#checkpoint(String,
String)
+ */
+ @Override
+ public Revision checkpoint(String pattern, String commitMessage,
AuthenticationInfo subject) {
+Revision revision = Revision.EMPTY;
+try {
+ List gitDiff = git.diff().call();
+ if (!gitDiff.isEmpty()) {
+LOG.debug("Changes found for pattern '{}': {}", pattern, gitDiff);
+DirCache added = git.add().addFilepattern(pattern).call();
+LOG.debug("{} changes are about to be commited",
added.getEntryCount());
+RevCommit commit = git.
[1/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
Repository: zeppelin
Updated Branches:
refs/heads/master db716c8b1 -> 3eea57ab2
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/NotebookRepoSyncTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/NotebookRepoSyncTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/NotebookRepoSyncTest.java
deleted file mode 100644
index 5571230..000
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/repo/NotebookRepoSyncTest.java
+++ /dev/null
@@ -1,437 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zeppelin.notebook.repo;
-
-import static com.google.common.truth.Truth.assertThat;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertTrue;
-import static org.mockito.Mockito.mock;
-
-import java.io.File;
-import java.io.IOException;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-
-import org.apache.commons.io.FileUtils;
-import org.apache.zeppelin.conf.ZeppelinConfiguration;
-import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
-import org.apache.zeppelin.dep.DependencyResolver;
-import org.apache.zeppelin.display.AngularObjectRegistryListener;
-import org.apache.zeppelin.helium.ApplicationEventListener;
-import org.apache.zeppelin.interpreter.InterpreterFactory;
-import org.apache.zeppelin.interpreter.InterpreterResultMessage;
-import org.apache.zeppelin.interpreter.InterpreterSettingManager;
-import org.apache.zeppelin.interpreter.remote.RemoteInterpreterProcessListener;
-import org.apache.zeppelin.notebook.*;
-import org.apache.zeppelin.scheduler.Job;
-import org.apache.zeppelin.scheduler.Job.Status;
-import org.apache.zeppelin.scheduler.SchedulerFactory;
-import org.apache.zeppelin.search.SearchService;
-import org.apache.zeppelin.storage.ConfigStorage;
-import org.apache.zeppelin.user.AuthenticationInfo;
-import org.apache.zeppelin.user.Credentials;
-import org.junit.After;
-import org.junit.Before;
-import org.junit.Test;
-import org.quartz.SchedulerException;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-
-public class NotebookRepoSyncTest implements JobListenerFactory {
-
- private File mainZepDir;
- private ZeppelinConfiguration conf;
- private SchedulerFactory schedulerFactory;
- private File mainNotebookDir;
- private File secNotebookDir;
- private Notebook notebookSync;
- private NotebookRepoSync notebookRepoSync;
- private InterpreterFactory factory;
- private InterpreterSettingManager interpreterSettingManager;
- private DependencyResolver depResolver;
- private SearchService search;
- private NotebookAuthorization notebookAuthorization;
- private Credentials credentials;
- private AuthenticationInfo anonymous;
- private static final Logger LOG =
LoggerFactory.getLogger(NotebookRepoSyncTest.class);
-
- @Before
- public void setUp() throws Exception {
-String zpath =
System.getProperty("java.io.tmpdir")+"/ZeppelinLTest_"+System.currentTimeMillis();
-mainZepDir = new File(zpath);
-mainZepDir.mkdirs();
-new File(mainZepDir, "conf").mkdirs();
-String mainNotePath = zpath+"/notebook";
-String secNotePath = mainNotePath + "_secondary";
-mainNotebookDir = new File(mainNotePath);
-secNotebookDir = new File(secNotePath);
-mainNotebookDir.mkdirs();
-secNotebookDir.mkdirs();
-
-System.setProperty(ConfVars.ZEPPELIN_HOME.getVarName(),
mainZepDir.getAbsolutePath());
-System.setProperty(ConfVars.ZEPPELIN_NOTEBOOK_DIR.getVarName(),
mainNotebookDir.getAbsolutePath());
-System.setProperty(ConfVars.ZEPPELIN_NOTEBOOK_STORAGE.getVarName(),
"org.apache.zeppelin.notebook.repo.VFSNotebookRepo,org.apache.zeppelin.notebook.repo.mock.VFSNotebookRepoMock");
-System.setProperty(ConfVars.ZEPPELIN_NOTEBOOK_ONE_WAY_SYNC.getVarName(),
"false");
-System.setProperty(ConfVars.ZEPPELIN_CONFIG_FS_DIR.getVarName(),
mainZepDir.getAbsolutePath() + "/conf");
-
-LOG.info("main Note dir : " + mainNotePath);
-LOG.info("secondary note dir : "
[7/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
ZEPPELIN-3196. Plugin framework for Zeppelin Engine ### What is this PR for? More details is in the design doc. https://docs.google.com/document/d/1QRgznfDKsi6XxR9FTJtUuiGAvgQvmc12FdIbHrmOZzc/edit?usp=sharing ### What type of PR is it? [ Improvement | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3196 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2760 from zjffdu/ZEPPELIN-3196 and squashes the following commits: 639c599 [Jeff Zhang] ZEPPELIN-3196. Plugin framework for Zeppelin Engine Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/3eea57ab Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/3eea57ab Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/3eea57ab Branch: refs/heads/master Commit: 3eea57ab26e8c4a8811bde64854b95f706858f10 Parents: db716c8 Author: Jeff Zhang Authored: Wed Jan 31 19:46:44 2018 +0800 Committer: Jeff Zhang Committed: Thu Mar 22 21:19:25 2018 +0800 -- .travis.yml | 23 +- pom.xml | 11 + .../src/assemble/distribution.xml | 3 + .../zeppelin/conf/ZeppelinConfiguration.java| 9 + zeppelin-plugins/notebookrepo/azure/pom.xml | 71 +++ .../notebook/repo/AzureNotebookRepo.java| 216 + ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../notebookrepo/filesystem/pom.xml | 236 ++ .../notebook/repo/FileSystemNotebookRepo.java | 103 + ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../repo/FileSystemNotebookRepoTest.java| 102 + zeppelin-plugins/notebookrepo/gcs/pom.xml | 141 ++ .../zeppelin/notebook/repo/GCSNotebookRepo.java | 214 + ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../notebook/repo/GCSNotebookRepoTest.java | 235 ++ zeppelin-plugins/notebookrepo/git/pom.xml | 71 +++ .../zeppelin/notebook/repo/GitNotebookRepo.java | 205 + ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../notebook/repo/GitNotebookRepoTest.java | 385 .../NotebookRepoSyncInitializationTest.java | 161 +++ .../notebook/repo/NotebookRepoSyncTest.java | 444 +++ .../notebook/repo/mock/VFSNotebookRepoMock.java | 36 ++ .../git/src/test/resources/2A94M5J1Z/note.json | 341 ++ .../git/src/test/resources/2A94M5J2Z/note.json | 87 .../git/src/test/resources/log4j.properties | 50 +++ zeppelin-plugins/notebookrepo/github/pom.xml| 50 +++ .../notebook/repo/GitHubNotebookRepo.java | 126 ++ ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../notebook/repo/GitHubNotebookRepoTest.java | 207 + .../src/test/resources/2A94M5J1Z/note.json | 341 ++ .../src/test/resources/2A94M5J2Z/note.json | 87 zeppelin-plugins/notebookrepo/mongodb/pom.xml | 55 +++ .../notebook/repo/MongoNotebookRepo.java| 222 ++ ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + zeppelin-plugins/notebookrepo/s3/pom.xml| 51 +++ .../zeppelin/notebook/repo/S3NotebookRepo.java | 294 ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + zeppelin-plugins/notebookrepo/vfs/pom.xml | 58 +++ .../zeppelin/notebook/repo/VFSNotebookRepo.java | 286 ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../notebook/repo/TestVFSNotebookRepo.java | 113 + .../notebookrepo/zeppelin-hub/pom.xml | 81 .../repo/zeppelinhub/ZeppelinHubRepo.java | 385 ...g.apache.zeppelin.notebook.repo.NotebookRepo | 18 + .../repo/zeppelinhub/ZeppelinHubRepoTest.java | 155 +++ .../websocket/ZeppelinClientTest.java | 127 ++ .../websocket/ZeppelinhubClientTest.java| 72 +++ .../websocket/mock/MockEchoWebsocketServer.java | 46 ++ .../websocket/mock/MockEventServlet.java| 14 + .../websocket/mock/MockEventSocket.java | 38 ++ .../protocol/ZeppelinhubMessageTest.java| 43 ++ zeppelin-plugins/pom.xml| 124 ++ zeppelin-server/pom.xml | 36 -- .../org/apache/zeppelin/realm/PamRealm.java | 2 +- .../apache/zeppelin/realm/ZeppelinHubRealm.java | 26 +- .../apache/zeppelin/socket/NotebookServer.java | 5 +- .../src/test/resources/log4j.properties | 3 +- zeppelin-zengine/pom.xml| 298 + .../recovery/FileSystemRecoveryStorage.java | 11 - .../java/org/apache/zeppelin/notebook/Note.java | 5
[5/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-plugins/notebookrepo/github/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java
--
diff --git
a/zeppelin-plugins/notebookrepo/github/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java
b/zeppelin-plugins/notebookrepo/github/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java
new file mode 100644
index 000..04a59ad
--- /dev/null
+++
b/zeppelin-plugins/notebookrepo/github/src/test/java/org/apache/zeppelin/notebook/repo/GitHubNotebookRepoTest.java
@@ -0,0 +1,207 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.notebook.repo;
+
+
+import com.google.common.base.Joiner;
+import org.apache.commons.io.FileUtils;
+import org.apache.zeppelin.conf.ZeppelinConfiguration;
+import org.apache.zeppelin.interpreter.InterpreterFactory;
+import org.apache.zeppelin.notebook.Note;
+import org.apache.zeppelin.notebook.Paragraph;
+import org.apache.zeppelin.user.AuthenticationInfo;
+import org.eclipse.jgit.api.Git;
+import org.eclipse.jgit.api.errors.GitAPIException;
+import org.eclipse.jgit.internal.storage.file.FileRepository;
+import org.eclipse.jgit.lib.Repository;
+import org.eclipse.jgit.revwalk.RevCommit;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Iterator;
+
+import static org.mockito.Mockito.mock;
+
+/**
+ * This tests the remote Git tracking for notebooks. The tests uses two local
Git repositories created locally
+ * to handle the tracking of Git actions (pushes and pulls). The repositories
are:
+ * 1. The first repository is considered as a remote that mimics a remote
GitHub directory
+ * 2. The second repository is considered as the local notebook repository
+ */
+public class GitHubNotebookRepoTest {
+ private static final Logger LOG =
LoggerFactory.getLogger(GitHubNotebookRepoTest.class);
+
+ private static final String TEST_NOTE_ID = "2A94M5J1Z";
+
+ private File remoteZeppelinDir;
+ private File localZeppelinDir;
+ private String localNotebooksDir;
+ private String remoteNotebooksDir;
+ private ZeppelinConfiguration conf;
+ private GitHubNotebookRepo gitHubNotebookRepo;
+ private RevCommit firstCommitRevision;
+ private Git remoteGit;
+
+ @Before
+ public void setUp() throws Exception {
+conf = ZeppelinConfiguration.create();
+
+String remoteRepositoryPath = System.getProperty("java.io.tmpdir") +
"/ZeppelinTestRemote_" +
+System.currentTimeMillis();
+String localRepositoryPath = System.getProperty("java.io.tmpdir") +
"/ZeppelinTest_" +
+System.currentTimeMillis();
+
+// Create a fake remote notebook Git repository locally in another
directory
+remoteZeppelinDir = new File(remoteRepositoryPath);
+remoteZeppelinDir.mkdirs();
+
+// Create a local repository for notebooks
+localZeppelinDir = new File(localRepositoryPath);
+localZeppelinDir.mkdirs();
+
+// Notebooks directory (for both the remote and local directories)
+localNotebooksDir = Joiner.on(File.separator).join(localRepositoryPath,
"notebook");
+remoteNotebooksDir = Joiner.on(File.separator).join(remoteRepositoryPath,
"notebook");
+
+File notebookDir = new File(localNotebooksDir);
+notebookDir.mkdirs();
+
+// Copy the test notebook directory from the test/resources/2A94M5J1Z
folder to the fake remote Git directory
+String remoteTestNoteDir =
Joiner.on(File.separator).join(remoteNotebooksDir, TEST_NOTE_ID);
+FileUtils.copyDirectory(
+new File(
+ GitHubNotebookRepoTest.class.getResource(
+Joiner.on(File.separator).join("", TEST_NOTE_ID)
+ ).getFile()
+), new File(remoteTestNoteDir)
+);
+
+// Create the fake remote Git repository
+Repository remoteRepository = new
FileRepository(Joiner.on(File.separator).join(remoteNotebooksDir, ".git"));
+remoteRepository.create();
+
+remoteGit = new Git(remoteRepositor
[2/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
--
diff --git
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
index 3b903e9..0fb9fb4 100644
---
a/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
+++
b/zeppelin-zengine/src/test/java/org/apache/zeppelin/notebook/NotebookTest.java
@@ -18,7 +18,6 @@
package org.apache.zeppelin.notebook;
import com.google.common.collect.Sets;
-import org.apache.commons.io.FileUtils;
import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
import org.apache.zeppelin.display.AngularObjectRegistry;
@@ -31,11 +30,10 @@ import org.apache.zeppelin.interpreter.InterpreterOption;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResultMessage;
import org.apache.zeppelin.interpreter.InterpreterSetting;
-import org.apache.zeppelin.notebook.repo.FileSystemNotebookRepo;
-import org.apache.zeppelin.notebook.repo.GitHubNotebookRepo;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreter;
import org.apache.zeppelin.notebook.repo.NotebookRepo;
-import org.apache.zeppelin.notebook.repo.VFSNotebookRepo;
+import org.apache.zeppelin.notebook.repo.NotebookRepoSettingsInfo;
+import org.apache.zeppelin.notebook.repo.NotebookRepoWithVersionControl;
import org.apache.zeppelin.resource.LocalResourcePool;
import org.apache.zeppelin.scheduler.Job;
import org.apache.zeppelin.scheduler.Job.Status;
@@ -46,10 +44,6 @@ import org.apache.zeppelin.user.Credentials;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
-import org.junit.runner.JUnitCore;
-import org.junit.runner.Request;
-import org.junit.runner.Result;
-import org.mockito.internal.runners.JUnit44RunnerImpl;
import org.quartz.SchedulerException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -57,6 +51,7 @@ import org.sonatype.aether.RepositoryException;
import java.io.File;
import java.io.IOException;
+import java.util.ArrayList;
import java.util.Arrays;
import java.util.Date;
import java.util.HashMap;
@@ -78,6 +73,7 @@ import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;
import static org.mockito.Mockito.mock;
+
public class NotebookTest extends AbstractInterpreterTest implements
JobListenerFactory {
private static final Logger logger =
LoggerFactory.getLogger(NotebookTest.class);
@@ -99,7 +95,7 @@ public class NotebookTest extends AbstractInterpreterTest
implements JobListener
schedulerFactory = SchedulerFactory.singleton();
SearchService search = mock(SearchService.class);
-notebookRepo = new VFSNotebookRepo(conf);
+notebookRepo = new InMemoryNotebookRepo();
notebookAuthorization = NotebookAuthorization.init(conf);
credentials = new Credentials(conf.credentialsPersist(),
conf.getCredentialsPath(), null);
@@ -117,26 +113,129 @@ public class NotebookTest extends
AbstractInterpreterTest implements JobListener
NotebookRepo notebookRepo;
Notebook notebook;
-notebookRepo = new VFSNotebookRepo(conf);
-notebook = new Notebook(conf, notebookRepo, schedulerFactory,
interpreterFactory,
-interpreterSettingManager, this, null,
-notebookAuthorization, credentials);
-assertFalse("Revision is not supported in VFSNotebookRepo",
notebook.isRevisionSupported());
-
-notebookRepo = new GitHubNotebookRepo(conf);
+notebookRepo = new DummyNotebookRepo();
notebook = new Notebook(conf, notebookRepo, schedulerFactory,
interpreterFactory,
interpreterSettingManager, this, null,
notebookAuthorization, credentials);
-assertTrue("Revision is supported in GitHubNotebookRepo",
notebook.isRevisionSupported());
+assertFalse("Revision is not supported in DummyNotebookRepo",
notebook.isRevisionSupported());
-notebookRepo = new FileSystemNotebookRepo(conf);
+notebookRepo = new DummyNotebookRepoWithVersionControl();
notebook = new Notebook(conf, notebookRepo, schedulerFactory,
interpreterFactory,
interpreterSettingManager, this, null,
notebookAuthorization, credentials);
-assertFalse("Revision is not supported in FileSystemNotebookRepo",
+assertTrue("Revision is supported in DummyNotebookRepoWithVersionControl",
notebook.isRevisionSupported());
}
+ public static class DummyNotebookRepo implements NotebookRepo {
+
+@Override
+public void init(ZeppelinConfiguration zConf) throws IOException {
+
+}
+
+@Override
+public List list(AuthenticationInfo subject) throws IOException {
+ return new ArrayList<>();
+}
+
+@Override
+public Note get(String noteId, Aut
[3/7] zeppelin git commit: ZEPPELIN-3196. Plugin framework for Zeppelin Engine
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/3eea57ab/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GCSNotebookRepo.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GCSNotebookRepo.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GCSNotebookRepo.java
deleted file mode 100644
index 6d62298..000
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/GCSNotebookRepo.java
+++ /dev/null
@@ -1,208 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- *http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.zeppelin.notebook.repo;
-
-import com.google.cloud.storage.Blob;
-import com.google.cloud.storage.BlobId;
-import com.google.cloud.storage.BlobInfo;
-import com.google.cloud.storage.Storage;
-import com.google.cloud.storage.Storage.BlobListOption;
-import com.google.cloud.storage.StorageException;
-import com.google.cloud.storage.StorageOptions;
-import com.google.common.base.Optional;
-import com.google.common.base.Preconditions;
-import com.google.common.base.Strings;
-import com.google.gson.JsonParseException;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-import java.util.Map;
-import java.util.regex.Matcher;
-import java.util.regex.Pattern;
-import org.apache.commons.lang.StringUtils;
-import org.apache.zeppelin.conf.ZeppelinConfiguration;
-import org.apache.zeppelin.conf.ZeppelinConfiguration.ConfVars;
-import org.apache.zeppelin.notebook.Note;
-import org.apache.zeppelin.notebook.NoteInfo;
-import org.apache.zeppelin.user.AuthenticationInfo;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-/**
- * A NotebookRepo implementation for storing notebooks in Google Cloud Storage.
- *
- * Notes are stored in the GCS "directory" specified by
zeppelin.notebook.gcs.dir. This path
- * must be in the form gs://bucketName/path/to/Dir. The bucket must already
exist. N.B: GCS is an
- * object store, so this "directory" should not itself be an object. Instead,
it represents the base
- * path for the note.json files.
- *
- * Authentication is provided by google-auth-library-java.
- * @see https://github.com/google/google-auth-library-java";>
- * google-auth-library-java.
- */
-public class GCSNotebookRepo implements NotebookRepo {
-
- private static final Logger LOG =
LoggerFactory.getLogger(GCSNotebookRepo.class);
- private String encoding;
- private String bucketName;
- private Optional basePath;
- private Pattern noteNamePattern;
- private Storage storage;
-
- public GCSNotebookRepo(ZeppelinConfiguration conf) throws IOException {
-this(conf, StorageOptions.getDefaultInstance().getService());
- }
-
- // For tests to use an in-memory storage implementation
- GCSNotebookRepo(ZeppelinConfiguration conf, Storage storage) throws
IOException {
-this.encoding = conf.getString(ConfVars.ZEPPELIN_ENCODING);
-
-String gcsStorageDir = conf.getGCSStorageDir();
-if (gcsStorageDir.isEmpty()) {
- throw new IOException("GCS storage directory must be set using
'zeppelin.notebook.gcs.dir'");
-}
-if (!gcsStorageDir.startsWith("gs://")) {
- throw new IOException(String.format(
- "GCS storage directory '%s' must start with 'gs://'.",
gcsStorageDir));
-}
-String storageDirWithoutScheme = gcsStorageDir.substring("gs://".length());
-
-// pathComponents excludes empty string if trailing slash is present
-List pathComponents =
Arrays.asList(storageDirWithoutScheme.split("/"));
-if (pathComponents.size() < 1) {
- throw new IOException(String.format(
- "GCS storage directory '%s' must be in the form
gs://bucketname/path/to/dir",
- gcsStorageDir));
-}
-this.bucketName = pathComponents.get(0);
-if (pathComponents.size() > 1) {
- this.basePath = Optional.of(StringUtils.join(
- pathComponents.subList(1, pathComponents.size()), "/"));
-} else {
- this.basePath = Optional.absent();
-}
-
-// Notes are stored at gs://bucketName/basePath//note.json
-if (basePath.isPresent()) {
- this.noteNamePatte
zeppelin git commit: ZEPPELIN-3355. Support inline configuration for user session
Repository: zeppelin
Updated Branches:
refs/heads/master 645037b36 -> a1e69add4
ZEPPELIN-3355. Support inline configuration for user session
### What is this PR for?
Generic ConfInterpreter only support process level configuration, this PR is
trying to introduce UserSessionConfInterpreter which can customize session
level configuration, One example is for livy interpreter that can customize
livy session.
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3355
### How should this be tested?
* Ci pass and manually test livy interpreter
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2883 from zjffdu/ZEPPELIN-3355 and squashes the following commits:
6708ef0 [Jeff Zhang] ZEPPELIN-3355. Support inline configuration for session
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/a1e69add
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/a1e69add
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/a1e69add
Branch: refs/heads/master
Commit: a1e69add442d112e00dbcf76b0abb21bc984e5be
Parents: 645037b
Author: Jeff Zhang
Authored: Tue Mar 20 17:35:13 2018 +0800
Committer: Jeff Zhang
Committed: Sun Mar 25 10:56:44 2018 +0800
--
.../zeppelin/interpreter/ConfInterpreter.java | 7 +-
.../interpreter/InterpreterSetting.java | 17 -
.../interpreter/InterpreterSettingManager.java | 4 --
.../interpreter/SessionConfInterpreter.java | 71
.../interpreter/remote/RemoteInterpreter.java | 6 ++
.../interpreter/SessionConfInterpreterTest.java | 69 +++
6 files changed, 166 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a1e69add/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
index d50c57b..7d1df9b 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
@@ -33,14 +33,17 @@ public class ConfInterpreter extends Interpreter {
private static Logger LOGGER =
LoggerFactory.getLogger(ConfInterpreter.class);
- private String interpreterGroupId;
- private InterpreterSetting interpreterSetting;
+ protected String sessionId;
+ protected String interpreterGroupId;
+ protected InterpreterSetting interpreterSetting;
public ConfInterpreter(Properties properties,
+ String sessionId,
String interpreterGroupId,
InterpreterSetting interpreterSetting) {
super(properties);
+this.sessionId = sessionId;
this.interpreterGroupId = interpreterGroupId;
this.interpreterSetting = interpreterSetting;
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/a1e69add/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index a3b7f8d..816499c 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -689,7 +689,16 @@ public class InterpreterSetting {
LOGGER.info("Interpreter {} created for user: {}, sessionId: {}",
interpreter.getClassName(), user, sessionId);
}
-interpreters.add(new ConfInterpreter(intpProperties, interpreterGroupId,
this));
+
+// TODO(zjffdu) this kind of hardcode is ugly. For now
SessionConfInterpreter is used
+// for livy, we could add new property in interpreter-setting.json when
there's new interpreter
+// require SessionConfInterpreter
+if (group.equals("livy")) {
+ interpreters.add(
+ new SessionConfInterpreter(intpProperties, sessionId,
interpreterGroupId, this));
+} else {
+ interpreters.add(new ConfInterpreter(intpProperties, sessionId,
interpreterGroupId, this));
+}
return interpreters;
}
@@ -751,7 +760,11 @@ public class InterpreterSetting {
//TODO(zjffdu) It requires use
zeppelin git commit: ZEPPELIN-3355. Support inline configuration for user session
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 f058f2a41 -> fbbb46b46
ZEPPELIN-3355. Support inline configuration for user session
### What is this PR for?
Generic ConfInterpreter only support process level configuration, this PR is
trying to introduce UserSessionConfInterpreter which can customize session
level configuration, One example is for livy interpreter that can customize
livy session.
### What type of PR is it?
[Feature ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3355
### How should this be tested?
* Ci pass and manually test livy interpreter
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2883 from zjffdu/ZEPPELIN-3355 and squashes the following commits:
6708ef0 [Jeff Zhang] ZEPPELIN-3355. Support inline configuration for session
(cherry picked from commit a1e69add442d112e00dbcf76b0abb21bc984e5be)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/fbbb46b4
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/fbbb46b4
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/fbbb46b4
Branch: refs/heads/branch-0.8
Commit: fbbb46b468b171e72232f5497e6d494d60f37ec3
Parents: f058f2a
Author: Jeff Zhang
Authored: Tue Mar 20 17:35:13 2018 +0800
Committer: Jeff Zhang
Committed: Sun Mar 25 10:56:57 2018 +0800
--
.../zeppelin/interpreter/ConfInterpreter.java | 7 +-
.../interpreter/InterpreterSetting.java | 17 -
.../interpreter/InterpreterSettingManager.java | 4 --
.../interpreter/SessionConfInterpreter.java | 71
.../interpreter/remote/RemoteInterpreter.java | 6 ++
.../interpreter/SessionConfInterpreterTest.java | 69 +++
6 files changed, 166 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/fbbb46b4/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
index d50c57b..7d1df9b 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/ConfInterpreter.java
@@ -33,14 +33,17 @@ public class ConfInterpreter extends Interpreter {
private static Logger LOGGER =
LoggerFactory.getLogger(ConfInterpreter.class);
- private String interpreterGroupId;
- private InterpreterSetting interpreterSetting;
+ protected String sessionId;
+ protected String interpreterGroupId;
+ protected InterpreterSetting interpreterSetting;
public ConfInterpreter(Properties properties,
+ String sessionId,
String interpreterGroupId,
InterpreterSetting interpreterSetting) {
super(properties);
+this.sessionId = sessionId;
this.interpreterGroupId = interpreterGroupId;
this.interpreterSetting = interpreterSetting;
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/fbbb46b4/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index a3b7f8d..816499c 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -689,7 +689,16 @@ public class InterpreterSetting {
LOGGER.info("Interpreter {} created for user: {}, sessionId: {}",
interpreter.getClassName(), user, sessionId);
}
-interpreters.add(new ConfInterpreter(intpProperties, interpreterGroupId,
this));
+
+// TODO(zjffdu) this kind of hardcode is ugly. For now
SessionConfInterpreter is used
+// for livy, we could add new property in interpreter-setting.json when
there's new interpreter
+// require SessionConfInterpreter
+if (group.equals("livy")) {
+ interpreters.add(
+ new SessionConfInterpreter(intpProperties, sessionId,
interpreterGroupId, this));
+} else {
+ interpreters.add(new ConfInterpreter(intpProperties, sessionId,
interpreterGroupId, this));
+}
return in
zeppelin git commit: ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter & IPythonInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master a1e69add4 -> 5f88452d6
ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter & IPythonInterpreter
### What is this PR for?
Unify the ZeppelinContext of PythonInterpreter & IPythonInterpreter into one
file to avoid code duplication.
### What type of PR is it?
[Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3362
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2890 from zjffdu/ZEPPELIN-3362 and squashes the following commits:
b5dcbc9 [Jeff Zhang] ZEPPELIN-3362. Unify ZeppelinContext of PythonInterpreter
& IPythonInterpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/5f88452d
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/5f88452d
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/5f88452d
Branch: refs/heads/master
Commit: 5f88452d63f9b6d7ad00328b1c5a0619e47cbc3c
Parents: a1e69ad
Author: Jeff Zhang
Authored: Fri Mar 23 17:23:59 2018 +0800
Committer: Jeff Zhang
Committed: Sun Mar 25 14:42:21 2018 +0800
--
.../zeppelin/python/IPythonInterpreter.java | 16 ++
.../zeppelin/python/PythonInterpreter.java | 9 +-
.../resources/grpc/python/zeppelin_python.py| 122 --
.../main/resources/python/zeppelin_context.py | 224 +++
.../main/resources/python/zeppelin_python.py| 182 +--
.../zeppelin/python/IPythonInterpreterTest.java | 10 +-
python/src/test/resources/log4j.properties | 1 +
.../zeppelin/spark/PySparkInterpreter.java | 10 +
.../main/resources/python/zeppelin_ipyspark.py | 6 +-
.../main/resources/python/zeppelin_pyspark.py | 168 ++
.../apache/zeppelin/plugin/PluginManager.java | 3 +-
11 files changed, 289 insertions(+), 462 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5f88452d/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java
index 5c5bfe3..4fe50ee 100644
--- a/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/IPythonInterpreter.java
@@ -210,6 +210,22 @@ public class IPythonInterpreter extends Interpreter
implements ExecuteResultHand
throw new IOException("Fail to setup JVMGateway\n" +
response.getOutput());
}
+input =
+
getClass().getClassLoader().getResourceAsStream("python/zeppelin_context.py");
+lines = IOUtils.readLines(input);
+response = ipythonClient.block_execute(ExecuteRequest.newBuilder()
+.setCode(StringUtils.join(lines, System.lineSeparator())).build());
+if (response.getStatus() == ExecuteStatus.ERROR) {
+ throw new IOException("Fail to import ZeppelinContext\n" +
response.getOutput());
+}
+
+response = ipythonClient.block_execute(ExecuteRequest.newBuilder()
+.setCode("z = __zeppelin__ =
PyZeppelinContext(intp.getZeppelinContext(), gateway)")
+.build());
+if (response.getStatus() == ExecuteStatus.ERROR) {
+ throw new IOException("Fail to setup ZeppelinContext\n" +
response.getOutput());
+}
+
if (additionalPythonInitFile != null) {
input =
getClass().getClassLoader().getResourceAsStream(additionalPythonInitFile);
lines = IOUtils.readLines(input);
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/5f88452d/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
--
diff --git
a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
index fab2ed9..178f79a 100644
--- a/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
+++ b/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java
@@ -65,6 +65,7 @@ import py4j.GatewayServer;
public class PythonInterpreter extends Interpreter implements
ExecuteResultHandler {
private static final Logger LOG =
LoggerFactory.getLogger(PythonInterpreter.class);
public static final String ZEPPELIN_PYTHON = "python/zeppelin_python.py";
+ public static final String ZEPPELIN_CONTEXT = "python/zeppelin_context.py";
public static final String ZE
zeppelin git commit: [ZEPPELIN-1967] Passing Z variables to Shell and SQL Interpreters
Repository: zeppelin
Updated Branches:
refs/heads/master 5f88452d6 -> 4e9d2c449
[ZEPPELIN-1967] Passing Z variables to Shell and SQL Interpreters
### What is this PR for?
The code in this PR enables embedding/interpolating Z variables into command
strings passed to Spark's SQL and Shell interpreters. It implements the
functionality described in issue
[ZEPPELIN-1967](https://issues.apache.org/jira/browse/ZEPPELIN-1967)
This PR resumes a fresh effort while taking into consideration all of the
discussion in the context of the earlier
[PR-2502](https://github.com/apache/zeppelin/pull/2502). The earlier PR-2502
was closed due to a corruption in my repo that could not be corrected.
The code in this PR resolves all of the discussion and suggestions in the body
of the earlier [PR-2502](https://github.com/apache/zeppelin/pull/2502). The
following description is a summary of the current implementation:
Patterns of the form `{var-name}` within commands will be interpolated only if
a predefined object of the specified name exists in `z`. Additionally, all such
patterns within the command line should also be translatable for any
interpolation to occur. Partial translation of a command line (where only some
of the patterns are translated, and others are not) is never performed.
Patterns of the form `{{any-text}}` are translated into `{any-text}`. This
feature is an escaping mechanism that allows `{` and `}` characters to be
passed into a command without invoking the interpolation mechanism.
The translations described above are performed only when all occurrences of
`{`, `}`, `{{`, and `}}` in any command string conform to one of the two forms
described above. A command that contains `{` and/or `}` characters used in any
other way (than `{var-name}` and `{{any-text}}` as described above) will be
used as-is without attempting any translations whatsoever -- even if the
command also contains legal, translate-able and/or escape-able, constructs of
the above two forms.
No error is flagged in any case. This behavior is identical to the
implementation of a similar feature in
Jupyter's shell invocation using the ! magic command.
At present only the SQL and Shell interpreters support object interpolation.
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-1967
### How should this be tested?
A new unit-test class ZeppCtxtVariableTest.java (see below) has been added. The
attached screenshots below also show tests of the functionality.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes, and detailed documentation has been added
to the part describing ZeppelinContext variables (see file spark.md below).
Author: Sanjay Dasgupta
Closes #2834 from sanjaydasgupta/ZEPPELIN-1967 and squashes the following
commits:
77738aa [Sanjay Dasgupta] Changes to comply with Felix Cheung's comment at
https://github.com/apache/zeppelin/pull/2834#discussion_r176976263 and Jeff
Zhang's subsequent clarification
5f8505b [Sanjay Dasgupta] Changes due to Felix Cheung's comments at
https://github.com/apache/zeppelin/pull/2834#pullrequestreview-106738198
d600d86 [Sanjay Dasgupta] Merge branch 'master' of
https://github.com/apache/zeppelin into ZEPPELIN-1967
cc3727f [Sanjay Dasgupta] Changes due the Jeff Zhang's comments at
https://github.com/apache/zeppelin/pull/2834/files/1e2c87dd36dc091ca898baf8e9f178d6d1a5e600#r176930418
1e2c87d [Sanjay Dasgupta] Merge branch 'master' of
https://github.com/apache/zeppelin into ZEPPELIN-1967
3dd3dd8 [Sanjay Dasgupta] Merge branch 'master' of
https://github.com/apache/zeppelin into ZEPPELIN-1967
a1703b8 [Sanjay Dasgupta] Changes suggested in Felix Cheung's review
https://github.com/apache/zeppelin/pull/2834#pullrequestreview-104805661
b7ddf6b [Sanjay Dasgupta] Implementing configuration (global enable/disable
interpolation) following
https://github.com/apache/zeppelin/pull/2834#issuecomment-373948398
5268803 [Sanjay Dasgupta] Merge branch 'master' of
https://github.com/apache/zeppelin into ZEPPELIN-1967
1718e79 [Sanjay Dasgupta] Merge branch 'master' of
https://github.com/apache/zeppelin into ZEPPELIN-1967
3b30ea2 [Sanjay Dasgupta] Reversing previous incorrect update
3beebce [Sanjay Dasgupta] Merge branch
zeppelin git commit: [ZEPPELIN-1967] Passing Z variables to Shell and SQL Interpreters
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 fbbb46b46 -> 726f82670
[ZEPPELIN-1967] Passing Z variables to Shell and SQL Interpreters
The code in this PR enables embedding/interpolating Z variables into command
strings passed to Spark's SQL and Shell interpreters. It implements the
functionality described in issue
[ZEPPELIN-1967](https://issues.apache.org/jira/browse/ZEPPELIN-1967)
This PR resumes a fresh effort while taking into consideration all of the
discussion in the context of the earlier
[PR-2502](https://github.com/apache/zeppelin/pull/2502). The earlier PR-2502
was closed due to a corruption in my repo that could not be corrected.
The code in this PR resolves all of the discussion and suggestions in the
body of the earlier [PR-2502](https://github.com/apache/zeppelin/pull/2502).
The following description is a summary of the current implementation:
Patterns of the form `{var-name}` within commands will be interpolated only
if a predefined object of the specified name exists in `z`. Additionally, all
such patterns within the command line should also be translatable for any
interpolation to occur. Partial translation of a command line (where only some
of the patterns are translated, and others are not) is never performed.
Patterns of the form `{{any-text}}` are translated into `{any-text}`. This
feature is an escaping mechanism that allows `{` and `}` characters to be
passed into a command without invoking the interpolation mechanism.
The translations described above are performed only when all occurrences of
`{`, `}`, `{{`, and `}}` in any command string conform to one of the two forms
described above. A command that contains `{` and/or `}` characters used in any
other way (than `{var-name}` and `{{any-text}}` as described above) will be
used as-is without attempting any translations whatsoever -- even if the
command also contains legal, translate-able and/or escape-able, constructs of
the above two forms.
No error is flagged in any case. This behavior is identical to the
implementation of a similar feature in
Jupyter's shell invocation using the ! magic command.
At present only the SQL and Shell interpreters support object interpolation.
[Improvement]
* [ ] - Task
https://issues.apache.org/jira/browse/ZEPPELIN-1967
A new unit-test class ZeppCtxtVariableTest.java (see below) has been added.
The attached screenshots below also show tests of the functionality.
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? Yes, and detailed documentation has been
added to the part describing ZeppelinContext variables (see file spark.md
below).
Author: Sanjay Dasgupta
Closes #2898 from zjffdu/ZEPPELIN-1967-0.8 and squashes the following commits:
800e3c5 [Sanjay Dasgupta] [ZEPPELIN-1967] Passing Z variables to Shell and SQL
Interpreters
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/726f8267
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/726f8267
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/726f8267
Branch: refs/heads/branch-0.8
Commit: 726f82670ab89299ee6a5824a63a2abaaaef557f
Parents: fbbb46b
Author: Sanjay Dasgupta
Authored: Mon Mar 26 13:31:09 2018 +0530
Committer: Jeff Zhang
Committed: Wed Mar 28 09:56:11 2018 +0800
--
docs/interpreter/shell.md | 29 ++-
docs/interpreter/spark.md | 54 +
.../apache/zeppelin/shell/ShellInterpreter.java | 4 +-
.../src/main/resources/interpreter-setting.json | 7 +
.../zeppelin/spark/SparkSqlInterpreter.java | 5 +-
.../src/main/resources/interpreter-setting.json | 7 +
.../zeppelin/interpreter/Interpreter.java | 34 ++-
.../interpreter/ZeppCtxtVariableTest.java | 212 +++
8 files changed, 348 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/726f8267/docs/interpreter/shell.md
--
diff --git a/docs/interpreter/shell.md b/docs/interpreter/shell.md
index d285cf4..2
zeppelin git commit: ZEPPELIN-3144. Fixed Checkstyle issues in markdown module
Repository: zeppelin
Updated Branches:
refs/heads/master 4e9d2c449 -> 930907959
ZEPPELIN-3144. Fixed Checkstyle issues in markdown module
### What is this PR for?
Fixed the Checkstyle issues in the markdown module.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3144
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Jan Hentschel
Closes #2897 from HorizonNet/ZEPPELIN-3144 and squashes the following commits:
1503c04 [Jan Hentschel] ZEPPELIN-3144. Fixed Checkstyle issues in markdown
module
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/93090795
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/93090795
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/93090795
Branch: refs/heads/master
Commit: 93090795942ddc331b2f4cb3ec6aac4fd7be55d5
Parents: 4e9d2c4
Author: Jan Hentschel
Authored: Tue Mar 27 13:39:47 2018 +0200
Committer: Jeff Zhang
Committed: Thu Mar 29 10:16:27 2018 +0800
--
markdown/pom.xml| 10 ++-
.../org/apache/zeppelin/markdown/Markdown.java | 9 ++-
.../zeppelin/markdown/Markdown4jParser.java | 2 -
.../markdown/PegdownWebSequencelPlugin.java | 23 +++
.../zeppelin/markdown/PegdownYumlPlugin.java| 25 ---
.../zeppelin/markdown/Markdown4jParserTest.java | 10 +--
.../zeppelin/markdown/PegdownParserTest.java| 71 +---
7 files changed, 86 insertions(+), 64 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/93090795/markdown/pom.xml
--
diff --git a/markdown/pom.xml b/markdown/pom.xml
index 4feb3bc..79cabc1 100644
--- a/markdown/pom.xml
+++ b/markdown/pom.xml
@@ -93,7 +93,13 @@
maven-resources-plugin
+
+org.apache.maven.plugins
+maven-checkstyle-plugin
+
+ false
+
+
-
-
+
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/93090795/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
--
diff --git a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
index 45fa6ce..83b4069 100644
--- a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown.java
@@ -17,21 +17,20 @@
package org.apache.zeppelin.markdown;
-import java.io.IOException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
import java.util.List;
import java.util.Properties;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
-import org.apache.zeppelin.interpreter.InterpreterPropertyBuilder;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.InterpreterResult.Code;
import org.apache.zeppelin.interpreter.InterpreterUtils;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.scheduler.Scheduler;
import org.apache.zeppelin.scheduler.SchedulerFactory;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
/**
* MarkdownInterpreter interpreter for Zeppelin.
@@ -74,7 +73,7 @@ public class Markdown extends Interpreter {
if (MarkdownParserType.PEGDOWN.toString().equals(parserType)) {
return new PegdownParser();
} else {
- /** default parser. */
+ // default parser
return new Markdown4jParser();
}
}
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/93090795/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java
--
diff --git
a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java
b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java
index 78f8137..215540d 100644
--- a/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java
+++ b/markdown/src/main/java/org/apache/zeppelin/markdown/Markdown4jParser.java
@@ -18,8 +18,6 @@
package org.apache.zeppelin.markdown;
import org.markdown4j.Markdown4jProcessor;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import java.io.IOException;
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/93090795/markdown/src/main/java/org/apache/zeppelin/markdown/PegdownWebSequencelPlugin.java
zeppelin git commit: ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve ZeppelinContext & add more test
Repository: zeppelin
Updated Branches:
refs/heads/master fd27014b0 -> b9d6056b7
ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve ZeppelinContext & add more
test
### What is this PR for?
This PR address the remaining issue of ZEPPELIN-3362, and also add more test
for ZeppelinContext.
### What type of PR is it?
[ Improvement ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3364
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2894 from zjffdu/ZEPPELIN-3364 and squashes the following commits:
891f1e1 [Jeff Zhang] ZEPPELIN-3364. Followup of ZEPPELIN-3362, improve
ZeppelinContext & add more test
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b9d6056b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b9d6056b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b9d6056b
Branch: refs/heads/master
Commit: b9d6056b7daa58d3021b70da2cfd8ad21c0ffb6e
Parents: fd27014
Author: Jeff Zhang
Authored: Sun Mar 25 22:13:44 2018 +0800
Committer: Jeff Zhang
Committed: Fri Mar 30 12:05:47 2018 +0800
--
.../main/resources/python/zeppelin_context.py | 26 -
.../zeppelin/python/IPythonInterpreterTest.java | 24 -
.../zeppelin/spark/IPySparkInterpreterTest.java | 2 +-
.../zeppelin/rest/ZeppelinSparkClusterTest.java | 57 +++-
4 files changed, 92 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b9d6056b/python/src/main/resources/python/zeppelin_context.py
--
diff --git a/python/src/main/resources/python/zeppelin_context.py
b/python/src/main/resources/python/zeppelin_context.py
index d97a789..d29a16f 100644
--- a/python/src/main/resources/python/zeppelin_context.py
+++ b/python/src/main/resources/python/zeppelin_context.py
@@ -17,6 +17,7 @@
import os, sys
import warnings
+import base64
from io import BytesIO
@@ -34,7 +35,7 @@ class PyZeppelinContext(object):
self.gateway = gateway
self.paramOption =
gateway.jvm.org.apache.zeppelin.display.ui.OptionInput.ParamOption
self.javaList = gateway.jvm.java.util.ArrayList
-self.max_result = 1000
+self.max_result = z.getMaxResult()
self._displayhook = lambda *args: None
self._setup_matplotlib()
@@ -129,13 +130,13 @@ class PyZeppelinContext(object):
# `isinstance(p, DataFrame)` would req `import
pandas.core.frame.DataFrame`
# and so a dependency on pandas
self.show_dataframe(p, **kwargs)
-elif hasattr(p, '__call__'):
-p() #error reporting
-
+else:
+print(str(p))
+
def show_dataframe(self, df, show_index=False, **kwargs):
"""Pretty prints DF using Table Display System
"""
-limit = len(df) > self.max_result
+exceed_limit = len(df) > self.max_result
header_buf = StringIO("")
if show_index:
idx_name = str(df.index.name) if df.index.name is not None else ""
@@ -147,7 +148,7 @@ class PyZeppelinContext(object):
header_buf.write("\n")
body_buf = StringIO("")
-rows = df.head(self.max_result).values if limit else df.values
+rows = df.head(self.max_result).values if exceed_limit else df.values
index = df.index.values
for idx, row in zip(index, rows):
if show_index:
@@ -158,13 +159,12 @@ class PyZeppelinContext(object):
body_buf.write("\t")
body_buf.write(str(cell))
body_buf.write("\n")
-body_buf.seek(0); header_buf.seek(0)
-#TODO(bzz): fix it, so it shows red notice, as in Spark
-print("%table " + header_buf.read() + body_buf.read()) # +
-# ("\nResults are limited by {}." \
-# .format(self.max_result) if limit else "")
-#)
+body_buf.seek(0)
+header_buf.seek(0)
+print("%table " + header_buf.read() + body_buf.read())
body_buf.close(); header_buf.close()
+if exceed_limit:
+print("%html Results are limited by
{}.".format(self.max_result))
def show_matplotlib(self, p, fmt="png", width="auto", height="auto",
**kwargs):
@@ -176,7 +176,7 @@ class PyZeppelinContext(object):
zeppelin git commit: trim property name of interpreter settings
Repository: zeppelin
Updated Branches:
refs/heads/master 0164cd90e -> b49b3f46f
trim property name of interpreter settings
### What is this PR for?
trim property name of interpreter settings
### What type of PR is it?
Improvement
### Todos
* [x] - Task
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-3335
### How should this be tested?
* First time? Setup Travis CI as described on
https://zeppelin.apache.org/contribution/contributions.html#continuous-integration
* Strongly recommended: add automated unit tests for any new or changed behavior
* Outline any manual steps to test the PR here.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?
Author: yunl.li
Closes #2884 from rockiee281/ZEPPELIN-3335-fix and squashes the following
commits:
f407a70 [yunl.li] trim interperter settings
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b49b3f46
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b49b3f46
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b49b3f46
Branch: refs/heads/master
Commit: b49b3f46f91d06c4d497f8f11cbc334fd480e539
Parents: 0164cd9
Author: yunl.li
Authored: Wed Mar 21 09:41:04 2018 +0800
Committer: Jeff Zhang
Committed: Sun Apr 1 11:26:17 2018 +0800
--
.../java/org/apache/zeppelin/interpreter/InterpreterSetting.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b49b3f46/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
index 816499c..ed8def0 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/InterpreterSetting.java
@@ -26,6 +26,7 @@ import com.google.gson.JsonObject;
import com.google.gson.annotations.SerializedName;
import com.google.gson.internal.StringMap;
import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
import org.apache.zeppelin.conf.ZeppelinConfiguration;
import org.apache.zeppelin.dep.Dependency;
import org.apache.zeppelin.dep.DependencyResolver;
@@ -494,7 +495,7 @@ public class InterpreterSetting {
StringMap map = (StringMap) properties;
Properties newProperties = new Properties();
for (String key : map.keySet()) {
-newProperties.put(key, map.get(key));
+newProperties.put(StringUtils.trim(key),
StringUtils.trim(map.get(key)));
}
this.properties = newProperties;
} else {
zeppelin git commit: ZEPPELIN-3344. Revert comments in queries in JDBC interpreter
Repository: zeppelin
Updated Branches:
refs/heads/master b49b3f46f -> 8238b711c
ZEPPELIN-3344. Revert comments in queries in JDBC interpreter
### What is this PR for?
The original purpose of https://github.com/apache/zeppelin/pull/2158 was
correct processing of ';'. This was done via full removing comments from code.
Unfortunately Apache Phoenix uses hooks in comments
https://forcedotcom.github.io/phoenix/#hintml.
Thus we should not delete comments in scripts.
There was discussion about comment rules for different databases (solr). The
right way is keep style. Thus analysts can copy queries to another tool and can
get same results and errors.
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3344
### How should this be tested?
* Unit tests pass: testSplitSqlQueryWithComments and testSplitSqlQuery
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: mebelousov
Closes #2876 from mebelousov/ZEPPELIN-3344 and squashes the following commits:
6980400 [mebelousov] ZEPPELIN-3344 Fix checkstyle, add tests for comments
83c8e8f [mebelousov] ZEPPELIN-3344 Rename test
eed54c8 [mebelousov] ZEPPELIN-3344 Revert comments in JDBC interpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/8238b711
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/8238b711
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/8238b711
Branch: refs/heads/master
Commit: 8238b711c7f6ff2c71ff807c6ee4ad52a29728a0
Parents: b49b3f4
Author: mebelousov
Authored: Tue Mar 20 08:57:02 2018 +0300
Committer: Jeff Zhang
Committed: Sun Apr 1 11:29:02 2018 +0800
--
.../apache/zeppelin/jdbc/JDBCInterpreter.java | 21 ++--
.../zeppelin/jdbc/JDBCInterpreterTest.java | 11 +++---
2 files changed, 14 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/8238b711/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
--
diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
index 56afe6f..a339059 100644
--- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
+++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
@@ -585,18 +585,12 @@ public class JDBCInterpreter extends KerberosInterpreter {
for (int item = 0; item < sql.length(); item++) {
character = sql.charAt(item);
- if ((singleLineComment && (character == '\n' || item == sql.length() -
1))
- || (multiLineComment && character == '/' && sql.charAt(item - 1) ==
'*')) {
+ if (singleLineComment && (character == '\n' || item == sql.length() -
1)) {
singleLineComment = false;
-multiLineComment = false;
-if (item == sql.length() - 1 && query.length() > 0) {
- queries.add(StringUtils.trim(query.toString()));
-}
-continue;
}
- if (singleLineComment || multiLineComment) {
-continue;
+ if (multiLineComment && character == '/' && sql.charAt(item - 1) == '*')
{
+multiLineComment = false;
}
if (character == '\'') {
@@ -619,16 +613,13 @@ public class JDBCInterpreter extends KerberosInterpreter {
&& sql.length() > item + 1) {
if (character == '-' && sql.charAt(item + 1) == '-') {
singleLineComment = true;
- continue;
-}
-
-if (character == '/' && sql.charAt(item + 1) == '*') {
+} else if (character == '/' && sql.charAt(item + 1) == '*') {
multiLineComment = true;
- continue;
}
}
- if (character == ';' && !quoteString && !doubleQuoteString) {
+ if (character == ';' && !quoteString && !doubleQuoteString &&
!multiLineComment
+ && !singleLineComment) {
queries.add(StringUtils.trim(query.toString()));
query = new StringBuilder();
} else if (item == sql.length() - 1) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/8238b711/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
--
diff --git
a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
index a6e394f..7a86a3d 100644
--- a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
+++ b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
@@ -189,13 +189,16 @@ public class JDBCInterpreterTest extends
BasicJDBCTestCaseAdapter {
zeppelin git commit: ZEPPELIN-3344. Revert comments in queries in JDBC interpreter
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 da277b9fb -> bbfae7c72
ZEPPELIN-3344. Revert comments in queries in JDBC interpreter
### What is this PR for?
The original purpose of https://github.com/apache/zeppelin/pull/2158 was
correct processing of ';'. This was done via full removing comments from code.
Unfortunately Apache Phoenix uses hooks in comments
https://forcedotcom.github.io/phoenix/#hintml.
Thus we should not delete comments in scripts.
There was discussion about comment rules for different databases (solr). The
right way is keep style. Thus analysts can copy queries to another tool and can
get same results and errors.
### What type of PR is it?
[Bug Fix]
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3344
### How should this be tested?
* Unit tests pass: testSplitSqlQueryWithComments and testSplitSqlQuery
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: mebelousov
Closes #2876 from mebelousov/ZEPPELIN-3344 and squashes the following commits:
6980400 [mebelousov] ZEPPELIN-3344 Fix checkstyle, add tests for comments
83c8e8f [mebelousov] ZEPPELIN-3344 Rename test
eed54c8 [mebelousov] ZEPPELIN-3344 Revert comments in JDBC interpreter
(cherry picked from commit 8238b711c7f6ff2c71ff807c6ee4ad52a29728a0)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/bbfae7c7
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/bbfae7c7
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/bbfae7c7
Branch: refs/heads/branch-0.8
Commit: bbfae7c725586e25eb1e71d60047b8804412208a
Parents: da277b9
Author: mebelousov
Authored: Tue Mar 20 08:57:02 2018 +0300
Committer: Jeff Zhang
Committed: Sun Apr 1 11:29:13 2018 +0800
--
.../apache/zeppelin/jdbc/JDBCInterpreter.java | 21 ++--
.../zeppelin/jdbc/JDBCInterpreterTest.java | 11 +++---
2 files changed, 14 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bbfae7c7/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
--
diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
index 0265e2d..fa69165 100644
--- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
+++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
@@ -588,18 +588,12 @@ public class JDBCInterpreter extends KerberosInterpreter {
for (int item = 0; item < sql.length(); item++) {
character = sql.charAt(item);
- if ((singleLineComment && (character == '\n' || item == sql.length() -
1))
- || (multiLineComment && character == '/' && sql.charAt(item - 1) ==
'*')) {
+ if (singleLineComment && (character == '\n' || item == sql.length() -
1)) {
singleLineComment = false;
-multiLineComment = false;
-if (item == sql.length() - 1 && query.length() > 0) {
- queries.add(StringUtils.trim(query.toString()));
-}
-continue;
}
- if (singleLineComment || multiLineComment) {
-continue;
+ if (multiLineComment && character == '/' && sql.charAt(item - 1) == '*')
{
+multiLineComment = false;
}
if (character == '\'') {
@@ -622,16 +616,13 @@ public class JDBCInterpreter extends KerberosInterpreter {
&& sql.length() > item + 1) {
if (character == '-' && sql.charAt(item + 1) == '-') {
singleLineComment = true;
- continue;
-}
-
-if (character == '/' && sql.charAt(item + 1) == '*') {
+} else if (character == '/' && sql.charAt(item + 1) == '*') {
multiLineComment = true;
- continue;
}
}
- if (character == ';' && !quoteString && !doubleQuoteString) {
+ if (character == ';' && !quoteString && !doubleQuoteString &&
!multiLineComment
+ && !singleLineComment) {
queries.add(StringUtils.trim(query.toString()));
query = new StringBuilder();
} else if (item == sql.length() - 1) {
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/bbfae7c7/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
--
diff --git
a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
index 6441267..f1587d5 100644
--- a/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpreterTest.java
+++ b/jdbc/src/test/java/org/apache/zeppelin/jdbc/JDBCInterpre
zeppelin git commit: ZEPPELIN-3374. Improvement on PySparkInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master 8238b711c -> aefc7ea39
ZEPPELIN-3374. Improvement on PySparkInterpreter
### What is this PR for?
A few improvements on PySparkInterpreter.
1. Refactor PySparkInterpreter to make it more readable
2. Code completion features is totally broken, fix it in this PR
3. Reuse the same test case of IPySparkInterpreter.
### What type of PR is it?
[Bug Fix | Improvement | Refactoring]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3374
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
Code completion before
Code completion after
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2901 from zjffdu/ZEPPELIN-3374 and squashes the following commits:
c22078d [Jeff Zhang] ZEPPELIN-3374. Improvement on PySparkInterpreter
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/aefc7ea3
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/aefc7ea3
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/aefc7ea3
Branch: refs/heads/master
Commit: aefc7ea395fed5a93e69942725a19b203a3574e2
Parents: 8238b71
Author: Jeff Zhang
Authored: Fri Mar 30 11:12:08 2018 +0800
Committer: Jeff Zhang
Committed: Sun Apr 1 14:31:12 2018 +0800
--
.../zeppelin/spark/PySparkInterpreter.java | 290 +--
.../main/resources/python/zeppelin_pyspark.py | 86 +++---
.../zeppelin/spark/IPySparkInterpreterTest.java | 127
.../zeppelin/spark/PySparkInterpreterTest.java | 39 +--
4 files changed, 259 insertions(+), 283 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/aefc7ea3/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index d97bb51..809e883 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -41,6 +41,7 @@ import
org.apache.zeppelin.interpreter.InterpreterResultMessage;
import org.apache.zeppelin.interpreter.InvalidHookException;
import org.apache.zeppelin.interpreter.LazyOpenInterpreter;
import org.apache.zeppelin.interpreter.WrappedInterpreter;
+import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.interpreter.util.InterpreterOutputStream;
import org.apache.zeppelin.spark.dep.SparkDependencyContext;
@@ -53,11 +54,8 @@ import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
-import java.io.OutputStreamWriter;
import java.io.PipedInputStream;
-import java.io.PipedOutputStream;
import java.net.MalformedURLException;
-import java.net.ServerSocket;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.LinkedList;
@@ -66,65 +64,26 @@ import java.util.Map;
import java.util.Properties;
/**
- *
+ * Interpreter for PySpark, it is the first implementation of interpreter for
PySpark, so with less
+ * features compared to IPySparkInterpreter, but requires less prerequisites
than
+ * IPySparkInterpreter, only python is required.
*/
public class PySparkInterpreter extends Interpreter implements
ExecuteResultHandler {
private static final Logger LOGGER =
LoggerFactory.getLogger(PySparkInterpreter.class);
+ private static final int MAX_TIMEOUT_SEC = 10;
+
private GatewayServer gatewayServer;
private DefaultExecutor executor;
- private int port;
+ // used to forward output from python process to InterpreterOutput
private InterpreterOutputStream outputStream;
- private BufferedWriter ins;
- private PipedInputStream in;
- private ByteArrayOutputStream input;
private String scriptPath;
- boolean pythonscriptRunning = false;
- private static final int MAX_TIMEOUT_SEC = 10;
- private long pythonPid;
-
+ private boolean pythonscriptRunning = false;
+ private long pythonPid = -1;
private IPySparkInterpreter iPySparkInterpreter;
+ private SparkInterpreter sparkInterpreter;
public PySparkInterpreter(Properties property) {
su
zeppelin git commit: [HOTFIX] JavaDoc Fix for branch-0.8
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 a7bf3ac4a -> 380a92768
[HOTFIX] JavaDoc Fix for branch-0.8
### What is this PR for?
Trivial fix for javadoc of branch-0.8
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
* No jira created
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang
Closes #2906 from zjffdu/JavaDoc_0.8 and squashes the following commits:
61ccd4d [Jeff Zhang] [HOTFIX] JavaDoc Fix for branch-0.8
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/380a9276
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/380a9276
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/380a9276
Branch: refs/heads/branch-0.8
Commit: 380a9276840342529ba335658ca9bf00657e79d6
Parents: a7bf3ac
Author: Jeff Zhang
Authored: Tue Apr 3 14:10:44 2018 +0800
Committer: Jeff Zhang
Committed: Tue Apr 3 21:38:15 2018 +0800
--
.../org/apache/zeppelin/interpreter/SessionConfInterpreter.java | 3 +++
1 file changed, 3 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/380a9276/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/SessionConfInterpreter.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/SessionConfInterpreter.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/SessionConfInterpreter.java
index e303ee6..f1fcdb4 100644
---
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/SessionConfInterpreter.java
+++
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/interpreter/SessionConfInterpreter.java
@@ -27,6 +27,9 @@ import java.io.StringReader;
import java.util.List;
import java.util.Properties;
+/**
+ * ConfInterpreter for session level
+ */
public class SessionConfInterpreter extends ConfInterpreter {
private static Logger LOGGER =
LoggerFactory.getLogger(SessionConfInterpreter.class);
zeppelin git commit: Hotfix: ambiguous reference to overloaded definition
Repository: zeppelin Updated Branches: refs/heads/master 881846364 -> 7a9572d67 Hotfix: ambiguous reference to overloaded definition ### What is this PR for? Minor bugfix on Zeppelin build from sources ### What type of PR is it? Bug Fix ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3324 ### How should this be tested? * mvn clean package \ --batch-mode package \ -Phelium-dev \ -Pscala-2.11 \ -Dscala.version=2.11.8 \ -Dscala.binary.version=2.11 \ -Pbuild-distr \ -Pspark-2.1 \ -Dspark.version=2.1.1 \ -Pr \ -Pcassandra-spark-1.5 \ -Psparkr \ -Ppyspark \ -Phadoop-2.7 \ -Dhadoop.version=2.7.3 \ -Dmaven.findbugs.enable=false \ -Drat.skip=true \ -Dcheckstyle.skip=true \ -DskipTests ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? N * Is there breaking changes for older versions? N * Does this needs documentation? N Author: Nelson Costa Closes #2863 from necosta/ZEPPELIN-3324 and squashes the following commits: a0de334 [Nelson Costa] Hotfix: ambiguous reference to overloaded definition Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7a9572d6 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7a9572d6 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7a9572d6 Branch: refs/heads/master Commit: 7a9572d67e4f02c6ad49ac091d6bc542732c19fc Parents: 8818463 Author: Nelson Costa Authored: Wed Mar 7 14:58:26 2018 + Committer: Jeff Zhang Committed: Thu Apr 5 20:49:25 2018 +0800 -- .../scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a9572d6/spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala -- diff --git a/spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala b/spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala index 43aa864..9d371ba 100644 --- a/spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala +++ b/spark/scala-2.10/src/main/scala/org/apache/zeppelin/spark/SparkScala210Interpreter.scala @@ -67,7 +67,7 @@ class SparkScala210Interpreter(override val conf: SparkConf, settings.usejavacp.value = true settings.classpath.value = getUserJars.mkString(File.pathSeparator) Console.setOut(interpreterOutput) -sparkILoop = new SparkILoop(null, new JPrintWriter(Console.out, true)) +sparkILoop = new SparkILoop() setDeclaredField(sparkILoop, "settings", settings) callMethod(sparkILoop, "createInterpreter")
zeppelin git commit: ZEPPELIN-3367. Unnecessary plugins folder is copied to zeppelin parent folder
Repository: zeppelin Updated Branches: refs/heads/master 685eb9249 -> 7a67138a9 ZEPPELIN-3367. Unnecessary plugins folder is copied to zeppelin parent folder ### What is this PR for? Change pom file to avoid the Unnecessary plugins folder copying to zeppelin parent folder which sometimes will cause build failure if user don't have permission to write the zeppelin parent folder. ### What type of PR is it? [Bug Fix] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3367 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? NO * Does this needs documentation? No Author: Jeff Zhang Closes #2913 from zjffdu/ZEPPELIN-3367 and squashes the following commits: 8d7bb21 [Jeff Zhang] ZEPPELIN-3367. Unnecessary plugins folder is copied to zeppelin parent folder Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/7a67138a Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/7a67138a Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/7a67138a Branch: refs/heads/master Commit: 7a67138a9e80fb6b38bd7434034f021ac61a7d2d Parents: 685eb92 Author: Jeff Zhang Authored: Fri Mar 30 09:21:41 2018 +0800 Committer: Jeff Zhang Committed: Sun Apr 8 16:15:24 2018 +0800 -- zeppelin-plugins/notebookrepo/azure/pom.xml | 8 ++ .../notebookrepo/filesystem/pom.xml | 8 ++ zeppelin-plugins/notebookrepo/gcs/pom.xml | 10 +++ zeppelin-plugins/notebookrepo/git/pom.xml | 8 ++ zeppelin-plugins/notebookrepo/github/pom.xml| 9 ++ zeppelin-plugins/notebookrepo/mongodb/pom.xml | 8 ++ zeppelin-plugins/notebookrepo/s3/pom.xml| 8 ++ zeppelin-plugins/notebookrepo/vfs/pom.xml | 8 ++ .../notebookrepo/zeppelin-hub/pom.xml | 9 ++ zeppelin-plugins/pom.xml| 87 ++-- 10 files changed, 120 insertions(+), 43 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a67138a/zeppelin-plugins/notebookrepo/azure/pom.xml -- diff --git a/zeppelin-plugins/notebookrepo/azure/pom.xml b/zeppelin-plugins/notebookrepo/azure/pom.xml index f663de6..2969175 100644 --- a/zeppelin-plugins/notebookrepo/azure/pom.xml +++ b/zeppelin-plugins/notebookrepo/azure/pom.xml @@ -68,4 +68,12 @@ + + + + +maven-dependency-plugin + + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a67138a/zeppelin-plugins/notebookrepo/filesystem/pom.xml -- diff --git a/zeppelin-plugins/notebookrepo/filesystem/pom.xml b/zeppelin-plugins/notebookrepo/filesystem/pom.xml index 422b7b6..712122d 100644 --- a/zeppelin-plugins/notebookrepo/filesystem/pom.xml +++ b/zeppelin-plugins/notebookrepo/filesystem/pom.xml @@ -41,6 +41,14 @@ NotebookRepo/FileSystemNotebookRepo + + + +maven-dependency-plugin + + + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a67138a/zeppelin-plugins/notebookrepo/gcs/pom.xml -- diff --git a/zeppelin-plugins/notebookrepo/gcs/pom.xml b/zeppelin-plugins/notebookrepo/gcs/pom.xml index 15aa7c7..b741c97 100644 --- a/zeppelin-plugins/notebookrepo/gcs/pom.xml +++ b/zeppelin-plugins/notebookrepo/gcs/pom.xml @@ -43,6 +43,7 @@ NotebookRepo/GCSNotebookRepo + com.google.cloud @@ -138,4 +139,13 @@ + + + + + +maven-dependency-plugin + + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a67138a/zeppelin-plugins/notebookrepo/git/pom.xml -- diff --git a/zeppelin-plugins/notebookrepo/git/pom.xml b/zeppelin-plugins/notebookrepo/git/pom.xml index 6f4ce8c..6f75502 100644 --- a/zeppelin-plugins/notebookrepo/git/pom.xml +++ b/zeppelin-plugins/notebookrepo/git/pom.xml @@ -68,4 +68,12 @@ + + + + +maven-dependency-plugin + + + http://git-wip-us.apache.org/repos/asf/zeppelin/blob/7a67138a/zeppelin-plugins/notebookrepo/github/pom.xml -- diff --git a/zeppelin-plugins/notebookrepo/github/pom.xml b/zeppelin-plugins/notebookrepo/github/pom.xml index 3673efb..5d14003 100644 --- a/zeppelin
zeppelin git commit: [ZEPPELIN-3350] Don't allow set cronExecutionUser
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 d3de51562 -> 852826156
[ZEPPELIN-3350] Don't allow set cronExecutionUser
### What is this PR for?
This PR just does a quick fix this security issue.
1. Remove the setting cron user in frontend
2. Run the note via owner.
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3350
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2914 from zjffdu/ZEPPELIN-3350 and squashes the following commits:
63325d0 [Jeff Zhang] [ZEPPELIN-3350] Don't allow set cronExecutionUser
(cherry picked from commit 08c9ad9a6692405884ae2b2f730868030247762f)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/85282615
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/85282615
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/85282615
Branch: refs/heads/branch-0.8
Commit: 85282615681e895d477417e9f9b80efdb221bf2c
Parents: d3de515
Author: Jeff Zhang
Authored: Wed Apr 4 18:13:06 2018 +0800
Committer: Jeff Zhang
Committed: Tue Apr 10 15:52:05 2018 +0800
--
docs/usage/other_features/cron_scheduler.md | 2 +-
zeppelin-web/src/app/notebook/notebook-actionBar.html | 7 ---
.../main/java/org/apache/zeppelin/notebook/Note.java | 14 +-
3 files changed, 6 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/85282615/docs/usage/other_features/cron_scheduler.md
--
diff --git a/docs/usage/other_features/cron_scheduler.md
b/docs/usage/other_features/cron_scheduler.md
index c7fc284..7223045 100644
--- a/docs/usage/other_features/cron_scheduler.md
+++ b/docs/usage/other_features/cron_scheduler.md
@@ -41,7 +41,7 @@ You can set a cron schedule easily by clicking each option
such as `1m` and `5m`
You can set the cron schedule by filling in this form. Please see [Cron
Trigger
Tutorial](http://www.quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/crontrigger)
for the available cron syntax.
-### Cron executing user
+### Cron executing user (It is removed from 0.8 where it enforces the cron
execution user to be the note owner for security purpose)
You can set the cron executing user by filling in this form and press the
enter key.
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/85282615/zeppelin-web/src/app/notebook/notebook-actionBar.html
--
diff --git a/zeppelin-web/src/app/notebook/notebook-actionBar.html
b/zeppelin-web/src/app/notebook/notebook-actionBar.html
index 7559a87..78ca77e 100644
--- a/zeppelin-web/src/app/notebook/notebook-actionBar.html
+++ b/zeppelin-web/src/app/notebook/notebook-actionBar.html
@@ -281,13 +281,6 @@ limitations under the License.
-- Cron executing user (click enter in field to
submit)
-
-
-
- auto-restart interpreter on cron execution
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/85282615/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 8889a55..3c49eac 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -637,16 +637,10 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
}
/**
- * Run all paragraphs sequentially.
+ * Run all paragraphs sequentially. Only used for CronJob
*/
public synchronized void runAll() {
-String cronExecutingUser = (String) getConfig().get("cronExecutingUser");
-if (null == cronExecutingUser) {
- cronExecutingUser = "anonymous";
-}
-AuthenticationInfo authenticationInfo = new AuthenticationInfo();
-authenticationInfo.setUser(cronExecutingUser);
-runAll(authenticationInfo, true);
+runAll(null, true);
}
public void runAll(AuthenticationInfo authenticationInfo, boolean blocking) {
@@ -654,7 +648,
zeppelin git commit: [ZEPPELIN-3350] Don't allow set cronExecutionUser
Repository: zeppelin
Updated Branches:
refs/heads/master 36a5a9173 -> 08c9ad9a6
[ZEPPELIN-3350] Don't allow set cronExecutionUser
### What is this PR for?
This PR just does a quick fix this security issue.
1. Remove the setting cron user in frontend
2. Run the note via owner.
### What type of PR is it?
[Bug Fix ]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3350
### How should this be tested?
* Manually tested
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2914 from zjffdu/ZEPPELIN-3350 and squashes the following commits:
63325d0 [Jeff Zhang] [ZEPPELIN-3350] Don't allow set cronExecutionUser
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/08c9ad9a
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/08c9ad9a
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/08c9ad9a
Branch: refs/heads/master
Commit: 08c9ad9a6692405884ae2b2f730868030247762f
Parents: 36a5a91
Author: Jeff Zhang
Authored: Wed Apr 4 18:13:06 2018 +0800
Committer: Jeff Zhang
Committed: Tue Apr 10 15:51:53 2018 +0800
--
docs/usage/other_features/cron_scheduler.md | 2 +-
zeppelin-web/src/app/notebook/notebook-actionBar.html | 7 ---
.../main/java/org/apache/zeppelin/notebook/Note.java | 14 +-
3 files changed, 6 insertions(+), 17 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/08c9ad9a/docs/usage/other_features/cron_scheduler.md
--
diff --git a/docs/usage/other_features/cron_scheduler.md
b/docs/usage/other_features/cron_scheduler.md
index c7fc284..7223045 100644
--- a/docs/usage/other_features/cron_scheduler.md
+++ b/docs/usage/other_features/cron_scheduler.md
@@ -41,7 +41,7 @@ You can set a cron schedule easily by clicking each option
such as `1m` and `5m`
You can set the cron schedule by filling in this form. Please see [Cron
Trigger
Tutorial](http://www.quartz-scheduler.org/documentation/quartz-2.2.x/tutorials/crontrigger)
for the available cron syntax.
-### Cron executing user
+### Cron executing user (It is removed from 0.8 where it enforces the cron
execution user to be the note owner for security purpose)
You can set the cron executing user by filling in this form and press the
enter key.
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/08c9ad9a/zeppelin-web/src/app/notebook/notebook-actionBar.html
--
diff --git a/zeppelin-web/src/app/notebook/notebook-actionBar.html
b/zeppelin-web/src/app/notebook/notebook-actionBar.html
index 7559a87..78ca77e 100644
--- a/zeppelin-web/src/app/notebook/notebook-actionBar.html
+++ b/zeppelin-web/src/app/notebook/notebook-actionBar.html
@@ -281,13 +281,6 @@ limitations under the License.
-- Cron executing user (click enter in field to
submit)
-
-
-
- auto-restart interpreter on cron execution
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/08c9ad9a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
--
diff --git
a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
index 3728cd3..664e986 100644
--- a/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
+++ b/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Note.java
@@ -642,16 +642,10 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
}
/**
- * Run all paragraphs sequentially.
+ * Run all paragraphs sequentially. Only used for CronJob
*/
public synchronized void runAll() {
-String cronExecutingUser = (String) getConfig().get("cronExecutingUser");
-if (null == cronExecutingUser) {
- cronExecutingUser = "anonymous";
-}
-AuthenticationInfo authenticationInfo = new AuthenticationInfo();
-authenticationInfo.setUser(cronExecutingUser);
-runAll(authenticationInfo, true);
+runAll(null, true);
}
public void runAll(AuthenticationInfo authenticationInfo, boolean blocking) {
@@ -659,7 +653,9 @@ public class Note implements ParagraphJobListener,
JsonSerializable {
if (!p
zeppelin git commit: ZEPPELIN-3156. Fixed Checkstyle errors and warnings in groovy module
Repository: zeppelin
Updated Branches:
refs/heads/master 08c9ad9a6 -> 008fa369f
ZEPPELIN-3156. Fixed Checkstyle errors and warnings in groovy module
### What is this PR for?
Fixed the Checkstyle errors and warnings in the groovy module.
### What type of PR is it?
Improvement
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3156
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: Jan Hentschel
Closes #2798 from HorizonNet/ZEPPELIN-3156 and squashes the following commits:
445ec95 [Jan Hentschel] ZEPPELIN-3156. Fixed Checkstyle errors and warnings in
groovy module
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/008fa369
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/008fa369
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/008fa369
Branch: refs/heads/master
Commit: 008fa369f37534096f59598f8b5374bcf56924e5
Parents: 08c9ad9
Author: Jan Hentschel
Authored: Tue Feb 13 14:01:47 2018 +0100
Committer: Jeff Zhang
Committed: Wed Apr 11 09:11:50 2018 +0800
--
groovy/pom.xml | 7
.../org/apache/zeppelin/groovy/GObject.java | 38
.../zeppelin/groovy/GroovyInterpreter.java | 38 +++-
3 files changed, 43 insertions(+), 40 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/008fa369/groovy/pom.xml
--
diff --git a/groovy/pom.xml b/groovy/pom.xml
index 8de3365..9b95fc3 100644
--- a/groovy/pom.xml
+++ b/groovy/pom.xml
@@ -80,6 +80,13 @@
maven-resources-plugin
+
+org.apache.maven.plugins
+maven-checkstyle-plugin
+
+ false
+
+
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/008fa369/groovy/src/main/java/org/apache/zeppelin/groovy/GObject.java
--
diff --git a/groovy/src/main/java/org/apache/zeppelin/groovy/GObject.java
b/groovy/src/main/java/org/apache/zeppelin/groovy/GObject.java
index babda8f..9948b89 100644
--- a/groovy/src/main/java/org/apache/zeppelin/groovy/GObject.java
+++ b/groovy/src/main/java/org/apache/zeppelin/groovy/GObject.java
@@ -16,43 +16,37 @@
*/
package org.apache.zeppelin.groovy;
-
-import java.io.StringWriter;
-
import org.slf4j.Logger;
-import java.util.Properties;
+import java.io.StringWriter;
import java.util.Collection;
-import java.util.Map;
-import java.util.List;
import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
-import groovy.xml.MarkupBuilder;
import groovy.lang.Closure;
+import groovy.xml.MarkupBuilder;
-import org.apache.zeppelin.interpreter.InterpreterContext;
-import org.apache.zeppelin.interpreter.InterpreterContextRunner;
-
-import org.apache.zeppelin.display.AngularObjectRegistry;
+import org.apache.zeppelin.annotation.ZeppelinApi;
import org.apache.zeppelin.display.AngularObject;
+import org.apache.zeppelin.display.AngularObjectRegistry;
import org.apache.zeppelin.display.GUI;
import org.apache.zeppelin.display.ui.OptionInput.ParamOption;
-import org.apache.zeppelin.annotation.ZeppelinApi;
+import org.apache.zeppelin.interpreter.InterpreterContext;
+import org.apache.zeppelin.interpreter.InterpreterContextRunner;
import org.apache.zeppelin.interpreter.RemoteWorksController;
-import org.apache.zeppelin.interpreter.InterpreterException;
/**
* Groovy interpreter for Zeppelin.
*/
public class GObject extends groovy.lang.GroovyObjectSupport {
-
Logger log;
StringWriter out;
Properties props;
InterpreterContext interpreterContext;
Map bindings;
-
public GObject(Logger log, StringWriter out, Properties p,
InterpreterContext ctx,
Map bindings) {
this.log = log;
@@ -90,7 +84,7 @@ public class GObject extends groovy.lang.GroovyObjectSupport {
}
/**
- * returns gui object
+ * returns gui object.
*/
public GUI getGui() {
return interpreterContext.getGui();
@@ -182,9 +176,9 @@ public class GObject extends
groovy.lang.GroovyObjectSupport {
}
/**
- * starts or continues rendering table rows
+ * starts or continues rendering table rows.
*
- * @param obj: 1. List(rows) of List(columns) where first line is a header
+ * @param obj List(rows) of List(columns) where first line is a header
*/
public void table(Object obj) {
if (obj == null) {
@@ -294,7 +288,7 @@ public class GObject extends
groovy.lang.GroovyObjectSupport {
}
/**
-
[3/3] zeppelin git commit: ZEPPELIN-3375: Make PySparkInterpreter extends PythonInterpreter
ZEPPELIN-3375: Make PySparkInterpreter extends PythonInterpreter ### What is this PR for? This PR is trying to remove the code duplication between PySparkInterpreter and PythonInterpreter. So here's the main things this PR did: * PySparkInterpreter extends PythonInterpreter * PySparkInterpreterTest extends PythonInterpreterTest so that we can verify PySparkInterpreter can do whatever PythonInterpreter can do * Move interpreter/lib/python/backend_zinline.py and interpreter/lib/python/mpl_config.py into python module, so that python module can ship these resources together. ### What type of PR is it? [ Improvement | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * https://issues.apache.org/jira/browse/ZEPPELIN-3375 ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2919 from zjffdu/ZEPPELIN-3375 and squashes the following commits: 738c6c5 [Jeff Zhang] ZEPPELIN-3375. Make PySparkInterpreter extends PythonInterpreter Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/0a97446a Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/0a97446a Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/0a97446a Branch: refs/heads/master Commit: 0a97446a70f6294a3efb071bb9a70601f885840b Parents: 7aa94ce Author: Jeff Zhang Authored: Sun Apr 1 21:26:28 2018 +0800 Committer: Jeff Zhang Committed: Mon Apr 16 14:17:54 2018 +0800 -- interpreter/lib/python/backend_zinline.py | 317 interpreter/lib/python/mpl_config.py| 105 --- python/pom.xml | 25 - .../zeppelin/python/IPythonInterpreter.java | 25 +- .../zeppelin/python/PythonCondaInterpreter.java | 77 +- .../python/PythonDockerInterpreter.java | 19 +- .../zeppelin/python/PythonInterpreter.java | 760 ++- .../python/PythonInterpreterPandasSql.java | 2 +- .../main/resources/python/backend_zinline.py| 317 python/src/main/resources/python/mpl_config.py | 105 +++ .../main/resources/python/py4j-src-0.9.2.zip| Bin 0 -> 71355 bytes .../main/resources/python/zeppelin_python.py| 159 ++-- .../python/BasePythonInterpreterTest.java | 331 .../zeppelin/python/IPythonInterpreterTest.java | 383 ++ .../python/PythonCondaInterpreterTest.java | 6 +- .../python/PythonDockerInterpreterTest.java | 23 +- .../zeppelin/python/PythonInterpreterTest.java | 165 ++-- python/src/test/resources/log4j.properties | 17 +- spark/interpreter/pom.xml | 4 +- .../zeppelin/spark/IPySparkInterpreter.java | 16 +- .../zeppelin/spark/NewSparkInterpreter.java | 7 +- .../zeppelin/spark/PySparkInterpreter.java | 581 ++ .../main/resources/python/zeppelin_pyspark.py | 215 +- .../zeppelin/spark/IPySparkInterpreterTest.java | 87 ++- .../zeppelin/spark/OldSparkInterpreterTest.java | 4 +- .../zeppelin/spark/PySparkInterpreterTest.java | 169 ++--- .../zeppelin/spark/SparkRInterpreterTest.java | 16 +- .../src/test/resources/log4j.properties | 6 +- .../zeppelin/interpreter/InterpreterGroup.java | 13 + .../zeppelin/rest/ZeppelinSparkClusterTest.java | 17 +- .../interpreter/InterpreterSetting.java | 3 +- 31 files changed, 1678 insertions(+), 2296 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0a97446a/interpreter/lib/python/backend_zinline.py -- diff --git a/interpreter/lib/python/backend_zinline.py b/interpreter/lib/python/backend_zinline.py deleted file mode 100644 index 1c84699..000 --- a/interpreter/lib/python/backend_zinline.py +++ /dev/null @@ -1,317 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -#http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# This file provides a static (non-inte
[1/3] zeppelin git commit: ZEPPELIN-3375: Make PySparkInterpreter extends PythonInterpreter
Repository: zeppelin
Updated Branches:
refs/heads/master 7aa94ce93 -> 0a97446a7
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0a97446a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
index 809e883..beebd42 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PySparkInterpreter.java
@@ -30,6 +30,7 @@ import org.apache.commons.lang.StringUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SQLContext;
+import org.apache.zeppelin.interpreter.BaseZeppelinContext;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
import org.apache.zeppelin.interpreter.InterpreterException;
@@ -44,6 +45,8 @@ import org.apache.zeppelin.interpreter.WrappedInterpreter;
import org.apache.zeppelin.interpreter.remote.RemoteInterpreterUtils;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.interpreter.util.InterpreterOutputStream;
+import org.apache.zeppelin.python.IPythonInterpreter;
+import org.apache.zeppelin.python.PythonInterpreter;
import org.apache.zeppelin.spark.dep.SparkDependencyContext;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -68,56 +71,23 @@ import java.util.Properties;
* features compared to IPySparkInterpreter, but requires less prerequisites
than
* IPySparkInterpreter, only python is required.
*/
-public class PySparkInterpreter extends Interpreter implements
ExecuteResultHandler {
- private static final Logger LOGGER =
LoggerFactory.getLogger(PySparkInterpreter.class);
- private static final int MAX_TIMEOUT_SEC = 10;
+public class PySparkInterpreter extends PythonInterpreter {
+
+ private static Logger LOGGER =
LoggerFactory.getLogger(PySparkInterpreter.class);
- private GatewayServer gatewayServer;
- private DefaultExecutor executor;
- // used to forward output from python process to InterpreterOutput
- private InterpreterOutputStream outputStream;
- private String scriptPath;
- private boolean pythonscriptRunning = false;
- private long pythonPid = -1;
- private IPySparkInterpreter iPySparkInterpreter;
private SparkInterpreter sparkInterpreter;
public PySparkInterpreter(Properties property) {
super(property);
+this.useBuiltinPy4j = false;
}
@Override
public void open() throws InterpreterException {
-// try IPySparkInterpreter first
-iPySparkInterpreter = getIPySparkInterpreter();
-if (getProperty("zeppelin.pyspark.useIPython", "true").equals("true") &&
-StringUtils.isEmpty(
-
iPySparkInterpreter.checkIPythonPrerequisite(getPythonExec(getProperties() {
- try {
-iPySparkInterpreter.open();
-LOGGER.info("IPython is available, Use IPySparkInterpreter to replace
PySparkInterpreter");
-return;
- } catch (Exception e) {
-iPySparkInterpreter = null;
-LOGGER.warn("Fail to open IPySparkInterpreter", e);
- }
-}
+setProperty("zeppelin.python.useIPython",
getProperty("zeppelin.pyspark.useIPython", "true"));
-// reset iPySparkInterpreter to null as it is not available
-iPySparkInterpreter = null;
-LOGGER.info("IPython is not available, use the native
PySparkInterpreter\n");
-// Add matplotlib display hook
-InterpreterGroup intpGroup = getInterpreterGroup();
-if (intpGroup != null && intpGroup.getInterpreterHookRegistry() != null) {
- try {
-// just for unit test I believe (zjffdu)
-registerHook(HookType.POST_EXEC_DEV.getName(),
"__zeppelin__._displayhook()");
- } catch (InvalidHookException e) {
-throw new InterpreterException(e);
- }
-}
+// create SparkInterpreter in JVM side TODO(zjffdu) move to
SparkInterpreter
DepInterpreter depInterpreter = getDepInterpreter();
-
// load libraries from Dependency Interpreter
URL [] urls = new URL[0];
List urlList = new LinkedList<>();
@@ -159,474 +129,81 @@ public class PySparkInterpreter extends Interpreter
implements ExecuteResultHand
ClassLoader oldCl = Thread.currentThread().getContextClassLoader();
try {
URLClassLoader newCl = new URLClassLoader(urls, oldCl);
- LOGGER.info("urls:" + urls);
- for (URL url : urls) {
-LOGGER.info("url:" + url);
- }
Thread.currentThread().setContextClassLoader(newCl);
+ // create Python Process and JVM gateway
+
[2/3] zeppelin git commit: ZEPPELIN-3375: Make PySparkInterpreter extends PythonInterpreter
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0a97446a/python/src/main/resources/python/zeppelin_python.py
--
diff --git a/python/src/main/resources/python/zeppelin_python.py
b/python/src/main/resources/python/zeppelin_python.py
index 0b2d533..19fa220 100644
--- a/python/src/main/resources/python/zeppelin_python.py
+++ b/python/src/main/resources/python/zeppelin_python.py
@@ -15,24 +15,12 @@
# limitations under the License.
#
-import os, sys, getopt, traceback, json, re
+import os, sys, traceback, json, re
from py4j.java_gateway import java_import, JavaGateway, GatewayClient
-from py4j.protocol import Py4JJavaError, Py4JNetworkError
-import warnings
-import ast
-import traceback
-import warnings
-import signal
-import base64
-
-from io import BytesIO
-try:
-from StringIO import StringIO
-except ImportError:
-from io import StringIO
+from py4j.protocol import Py4JJavaError
-# for back compatibility
+import ast
class Logger(object):
def __init__(self):
@@ -47,46 +35,79 @@ class Logger(object):
def flush(self):
pass
-def handler_stop_signals(sig, frame):
- sys.exit("Got signal : " + str(sig))
+class PythonCompletion:
+ def __init__(self, interpreter, userNameSpace):
+self.interpreter = interpreter
+self.userNameSpace = userNameSpace
-signal.signal(signal.SIGINT, handler_stop_signals)
+ def getObjectCompletion(self, text_value):
+completions = [completion for completion in
list(self.userNameSpace.keys()) if completion.startswith(text_value)]
+builtinCompletions = [completion for completion in dir(__builtins__) if
completion.startswith(text_value)]
+return completions + builtinCompletions
-host = "127.0.0.1"
-if len(sys.argv) >= 3:
- host = sys.argv[2]
+ def getMethodCompletion(self, objName, methodName):
+execResult = locals()
+try:
+ exec("{} = dir({})".format("objectDefList", objName),
_zcUserQueryNameSpace, execResult)
+except:
+ self.interpreter.logPythonOutput("Fail to run dir on " + objName)
+ self.interpreter.logPythonOutput(traceback.format_exc())
+ return None
+else:
+ objectDefList = execResult['objectDefList']
+ return [completion for completion in execResult['objectDefList'] if
completion.startswith(methodName)]
+
+ def getCompletion(self, text_value):
+if text_value == None:
+ return None
+
+dotPos = text_value.find(".")
+if dotPos == -1:
+ objName = text_value
+ completionList = self.getObjectCompletion(objName)
+else:
+ objName = text_value[:dotPos]
+ methodName = text_value[dotPos + 1:]
+ completionList = self.getMethodCompletion(objName, methodName)
+
+if completionList is None or len(completionList) <= 0:
+ self.interpreter.setStatementsFinished("", False)
+else:
+ result = json.dumps(list(filter(lambda x : not re.match("^__.*", x),
list(completionList
+ self.interpreter.setStatementsFinished(result, False)
+
+host = sys.argv[1]
+port = int(sys.argv[2])
+
+client = GatewayClient(address=host, port=port)
+gateway = JavaGateway(client, auto_convert = True)
+intp = gateway.entry_point
+# redirect stdout/stderr to java side so that PythonInterpreter can capture
the python execution result
+output = Logger()
+sys.stdout = output
+sys.stderr = output
_zcUserQueryNameSpace = {}
-client = GatewayClient(address=host, port=int(sys.argv[1]))
-
-gateway = JavaGateway(client)
-
-intp = gateway.entry_point
-intp.onPythonScriptInitialized(os.getpid())
-java_import(gateway.jvm, "org.apache.zeppelin.display.Input")
+completion = PythonCompletion(intp, _zcUserQueryNameSpace)
+_zcUserQueryNameSpace["__zeppelin_completion__"] = completion
+_zcUserQueryNameSpace["gateway"] = gateway
from zeppelin_context import PyZeppelinContext
+if intp.getZeppelinContext():
+ z = __zeppelin__ = PyZeppelinContext(intp.getZeppelinContext(), gateway)
+ __zeppelin__._setup_matplotlib()
+ _zcUserQueryNameSpace["z"] = z
+ _zcUserQueryNameSpace["__zeppelin__"] = __zeppelin__
-z = __zeppelin__ = PyZeppelinContext(intp.getZeppelinContext(), gateway)
-__zeppelin__._setup_matplotlib()
-
-_zcUserQueryNameSpace["__zeppelin__"] = __zeppelin__
-_zcUserQueryNameSpace["z"] = z
-
-output = Logger()
-sys.stdout = output
-#sys.stderr = output
+intp.onPythonScriptInitialized(os.getpid())
while True :
req = intp.getStatements()
- if req == None:
-break
-
try:
stmts = req.statements().split("\n")
-final_code = []
+isForCompletion = req.isForCompletion()
# Get post-execute hooks
try:
@@ -98,35 +119,23 @@ while True :
user_hook = __zeppelin__.getHook('post_exec')
except:
user_hook = None
-
-nhooks = 0
-for hook in (global_hook, user_hook):
- if hook:
-nhooks += 1
-for s in stmts:
- if s == None:
-continue
-
- # skip comment
- s_stripped = s.strip()
- if len(s_s
zeppelin git commit: [ZEPPELIN-2679] JDBC. precode for session
Repository: zeppelin
Updated Branches:
refs/heads/master 0a97446a7 -> edf3c2589
[ZEPPELIN-2679] JDBC. precode for session
### What is this PR for?
We have a precode for interpreter (all interpreters) which executes once after
start interpreter. With it we can create shared code that will be available in
the paragraph, create tables in the database and etc.
This precode for jdbc only which executes before run paragraph in same session
(sql). In this precode we can declare session variables that will be available
when you run paragraph (any sql which will be executed before the paragraph).
For example `set application_name='#{user};`
Previous discussion in #2442
### What type of PR is it?
Feature
### What is the Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-2679
### How should this be tested?
(jdbc interpreter, postgres)
1. set property **default.statementPrecode** `set application_name='#{user}';`
2. run paragraph `select current_setting('application_name');`
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? no
Author: tinkoff-dwh
Author: mebelousov
Author: Tinkoff DWH
Author: Renjith Kamath
Author: isys.mreshetov
Closes #2920 from mebelousov/ZEPPELIN-2679 and squashes the following commits:
d610805 [mebelousov] Revert "ZEPPELIN-3406 Travis build failure due to npm
error"
11cbc44 [Renjith Kamath] ZEPPELIN-3406 Travis build failure due to npm error
11f89a5 [mebelousov] ZEPPELIN-2679 Small fix in docs
e9d36d2 [mebelousov] Merge remote-tracking branch 'apache/master' into
ZEPPELIN-2679
00aeabf [tinkoff-dwh] ZEPPELIN-2679 fix checkstyle
f7529ca [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into
ZEPPELIN-2679
07b41c5 [tinkoff-dwh] ZEPPELIN-2679 rename test methods
cd0d6ed [tinkoff-dwh] Merge remote-tracking branch 'upstream/master' into
ZEPPELIN-2679
7bba0db [tinkoff-dwh] ZEPPELIN-2679 rename session to statement
719d0a2 [isys.mreshetov] ZEPPELIN-2679 docs
ae7a0d6 [Tinkoff DWH] [ZEPPELIN-2679] tests
bd3d809 [Tinkoff DWH] [ZEPPELIN-2679] jdbc. precode for session
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/edf3c258
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/edf3c258
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/edf3c258
Branch: refs/heads/master
Commit: edf3c2589b2d1ccc080e2b9b30dbbd9c53e9db54
Parents: 0a97446
Author: tinkoff-dwh
Authored: Sun Apr 15 21:16:55 2018 +0300
Committer: Jeff Zhang
Committed: Mon Apr 16 16:46:44 2018 +0800
--
docs/interpreter/jdbc.md| 7 +-
.../apache/zeppelin/jdbc/JDBCInterpreter.java | 10 +++
.../src/main/resources/interpreter-setting.json | 8 ++-
.../zeppelin/jdbc/JDBCInterpreterTest.java | 67 +++-
4 files changed, 88 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/edf3c258/docs/interpreter/jdbc.md
--
diff --git a/docs/interpreter/jdbc.md b/docs/interpreter/jdbc.md
index 44f8aab..a6c28df 100644
--- a/docs/interpreter/jdbc.md
+++ b/docs/interpreter/jdbc.md
@@ -125,6 +125,11 @@ The JDBC interpreter properties are defined by default
like below.
Some SQL which executes every time after initialization of the
interpreter (see Binding
mode)
+default.statementPrecode
+
+SQL code which executed before the SQL from paragraph, in the same
database session (database connection)
+
+
default.completer.schemaFilters
Сomma separated schema (schema = catalog = database) filters to get
metadata for completions. Supports '%' symbol is equivalent to any set of
characters. (ex. prod_v_%,public%,info)
@@ -306,7 +311,7 @@ Returns value of `search_path` which is set in the
*default.precode*.
```sql
-%jdbc(mysql)
+%mysql
select @v
```
Returns value of `v` which is set in the *mysql.precode*.
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/edf3c258/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
--
diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
index a339059..4bfaf04 100644
--- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
+++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
@@ -103,6 +103,7 @@ public class JDBCInterpreter extends KerberosInterpreter {
static final String USER_KEY = "user";
static final String PASSWORD_KEY = "password";
static final String PRECODE_KEY = "precode";
+ static final String STATEMENT_PRECODE_KEY = "statementPrecode";
static final String
