zeppelin git commit: ZEPPELIN-2221 Show `Jobs` page when `jobId` is missing
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 d7918ab8d -> d17535bca
ZEPPELIN-2221 Show `Jobs` page when `jobId` is missing
### What is this PR for?
Because of yarn's bug, Spark's `jobId` is not passed. This causes some Spark UI
link looks broken. In this kind of case, showing `Jobs` page looks reasonable.
Yarn's bug is already fixed with the latest version only, so we need to handle
it in Zeppelin side.
### What type of PR is it?
[Bug Fix]
### Todos
* [x] - Return `Jobs` page when `Job` page is not available
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2221
### How should this be tested?
1. Install outdated yarn
2. Run script
3. Click `Spark UI` to show `Jobs` page instead of 404
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee
Closes #2964 from jongyoul/ZEPPELIN-2221 and squashes the following commits:
d595028b5 [Jongyoul Lee] Remove unused packages from import statements Add
spark's master value as a parameter of setupSparkListener Add parameterized
test for various yarn versions Add a test checking jobUrl Add a logic to check
a hadoop version and yarn version to support passing `get` parameters Add
hadoop dependency as a `provided` scope
8b6f9076d [Jongyoul Lee] Fix to cover spark1 and spark2
8fdc46763 [Jongyoul Lee] Fix to cover spark1 and spark2
7ffcc55ca [Jongyoul Lee] Consume httpEntity
431457308 [Jongyoul Lee] Add httpClient to test if `jobUrl` is valid or not Add
test cases for that feature
(cherry picked from commit b6beda64e7d25fa93958737fc8d2dad4eae309cc)
Signed-off-by: Jongyoul Lee
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/d17535bc
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/d17535bc
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/d17535bc
Branch: refs/heads/branch-0.8
Commit: d17535bca7db8f41a037e8f8026bec4b221632e7
Parents: d7918ab
Author: Jongyoul Lee
Authored: Mon May 21 15:15:48 2018 +0900
Committer: Jongyoul Lee
Committed: Wed May 30 13:03:27 2018 +0900
--
.../zeppelin/spark/NewSparkInterpreter.java | 8 +-
.../zeppelin/spark/OldSparkInterpreter.java | 4 +-
.../apache/zeppelin/spark/SparkShimsTest.java | 153 +++
spark/spark-shims/pom.xml | 11 ++
.../org/apache/zeppelin/spark/SparkShims.java | 59 +--
.../org/apache/zeppelin/spark/Spark1Shims.java | 25 +--
.../org/apache/zeppelin/spark/Spark2Shims.java | 4 +-
7 files changed, 217 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/d17535bc/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
index 591ef96..2fa1093 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
@@ -23,10 +23,7 @@ import org.apache.commons.lang3.exception.ExceptionUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.spark.scheduler.SparkListenerJobStart;
import org.apache.spark.sql.SQLContext;
-import org.apache.spark.ui.jobs.JobProgressListener;
-import org.apache.zeppelin.interpreter.BaseZeppelinContext;
import org.apache.zeppelin.interpreter.DefaultInterpreterProperty;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
@@ -34,7 +31,6 @@ import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterHookRegistry;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.WrappedInterpreter;
-import org.apache.zeppelin.interpreter.remote.RemoteEventClientWrapper;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.spark.dep.SparkDependencyContext;
import org.slf4j.Logger;
@@ -42,8 +38,6 @@ import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.IOException;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -119,7 +113,7 @@ public class NewSparkInterpreter extends
AbstractSparkInterpreter {
sparkSession = this.innerInterpreter.sparkSession();
zeppelin git commit: ZEPPELIN-2221 Show `Jobs` page when `jobId` is missing
Repository: zeppelin
Updated Branches:
refs/heads/master 0c29a95bb -> b6beda64e
ZEPPELIN-2221 Show `Jobs` page when `jobId` is missing
### What is this PR for?
Because of yarn's bug, Spark's `jobId` is not passed. This causes some Spark UI
link looks broken. In this kind of case, showing `Jobs` page looks reasonable.
Yarn's bug is already fixed with the latest version only, so we need to handle
it in Zeppelin side.
### What type of PR is it?
[Bug Fix]
### Todos
* [x] - Return `Jobs` page when `Job` page is not available
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-2221
### How should this be tested?
1. Install outdated yarn
2. Run script
3. Click `Spark UI` to show `Jobs` page instead of 404
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jongyoul Lee
Closes #2964 from jongyoul/ZEPPELIN-2221 and squashes the following commits:
d595028b5 [Jongyoul Lee] Remove unused packages from import statements Add
spark's master value as a parameter of setupSparkListener Add parameterized
test for various yarn versions Add a test checking jobUrl Add a logic to check
a hadoop version and yarn version to support passing `get` parameters Add
hadoop dependency as a `provided` scope
8b6f9076d [Jongyoul Lee] Fix to cover spark1 and spark2
8fdc46763 [Jongyoul Lee] Fix to cover spark1 and spark2
7ffcc55ca [Jongyoul Lee] Consume httpEntity
431457308 [Jongyoul Lee] Add httpClient to test if `jobUrl` is valid or not Add
test cases for that feature
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b6beda64
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b6beda64
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b6beda64
Branch: refs/heads/master
Commit: b6beda64e7d25fa93958737fc8d2dad4eae309cc
Parents: 0c29a95
Author: Jongyoul Lee
Authored: Mon May 21 15:15:48 2018 +0900
Committer: Jongyoul Lee
Committed: Wed May 30 13:03:16 2018 +0900
--
.../zeppelin/spark/NewSparkInterpreter.java | 8 +-
.../zeppelin/spark/OldSparkInterpreter.java | 4 +-
.../apache/zeppelin/spark/SparkShimsTest.java | 153 +++
spark/spark-shims/pom.xml | 11 ++
.../org/apache/zeppelin/spark/SparkShims.java | 59 +--
.../org/apache/zeppelin/spark/Spark1Shims.java | 25 +--
.../org/apache/zeppelin/spark/Spark2Shims.java | 4 +-
7 files changed, 217 insertions(+), 47 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b6beda64/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
--
diff --git
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
index b2b715b..b77e1a7 100644
---
a/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
+++
b/spark/interpreter/src/main/java/org/apache/zeppelin/spark/NewSparkInterpreter.java
@@ -23,10 +23,7 @@ import org.apache.commons.lang3.exception.ExceptionUtils;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkContext;
import org.apache.spark.api.java.JavaSparkContext;
-import org.apache.spark.scheduler.SparkListenerJobStart;
import org.apache.spark.sql.SQLContext;
-import org.apache.spark.ui.jobs.JobProgressListener;
-import org.apache.zeppelin.interpreter.BaseZeppelinContext;
import org.apache.zeppelin.interpreter.DefaultInterpreterProperty;
import org.apache.zeppelin.interpreter.Interpreter;
import org.apache.zeppelin.interpreter.InterpreterContext;
@@ -34,7 +31,6 @@ import org.apache.zeppelin.interpreter.InterpreterException;
import org.apache.zeppelin.interpreter.InterpreterHookRegistry;
import org.apache.zeppelin.interpreter.InterpreterResult;
import org.apache.zeppelin.interpreter.WrappedInterpreter;
-import org.apache.zeppelin.interpreter.remote.RemoteEventClientWrapper;
import org.apache.zeppelin.interpreter.thrift.InterpreterCompletion;
import org.apache.zeppelin.spark.dep.SparkDependencyContext;
import org.slf4j.Logger;
@@ -42,8 +38,6 @@ import org.slf4j.LoggerFactory;
import java.io.File;
import java.io.IOException;
-import java.lang.reflect.InvocationTargetException;
-import java.lang.reflect.Method;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
@@ -119,7 +113,7 @@ public class NewSparkInterpreter extends
AbstractSparkInterpreter {
sparkSession = this.innerInterpreter.sparkSession();
sparkUrl = this.innerInterpreter.sparkUrl();
sparkShims = SparkShims.getInstance(sc.version());
-
svn commit: r27153 - /dev/zeppelin/zeppelin-0.8.0-rc4/
Author: zjffdu Date: Wed May 30 01:00:20 2018 New Revision: 27153 Log: Remove Zeppelin 0.8.0 RC4 Removed: dev/zeppelin/zeppelin-0.8.0-rc4/
[1/2] zeppelin git commit: Preparing Apache Zeppelin release 0.8.0
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 74ffc2134 -> d7918ab8d Preparing Apache Zeppelin release 0.8.0 Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/41d9e0ac Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/41d9e0ac Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/41d9e0ac Branch: refs/heads/branch-0.8 Commit: 41d9e0ac3f908da2eb4167984271aced78a8bfc6 Parents: 74ffc21 Author: Jeff Zhang Authored: Tue May 29 21:36:55 2018 +0800 Committer: Jeff Zhang Committed: Tue May 29 21:36:55 2018 +0800 -- alluxio/pom.xml | 4 ++-- angular/pom.xml | 4 ++-- beam/pom.xml| 4 ++-- bigquery/pom.xml| 4 ++-- cassandra/pom.xml | 4 ++-- elasticsearch/pom.xml | 4 ++-- file/pom.xml| 4 ++-- flink/pom.xml | 4 ++-- geode/pom.xml | 4 ++-- groovy/pom.xml | 4 ++-- hbase/pom.xml | 4 ++-- helium-dev/pom.xml | 4 ++-- ignite/pom.xml | 4 ++-- interpreter-parent/pom.xml | 4 ++-- jdbc/pom.xml| 4 ++-- kylin/pom.xml | 4 ++-- lens/pom.xml| 4 ++-- livy/pom.xml| 4 ++-- markdown/pom.xml| 4 ++-- neo4j/pom.xml | 4 ++-- pig/pom.xml | 4 ++-- pom.xml | 2 +- python/pom.xml | 4 ++-- r/pom.xml | 4 ++-- sap/pom.xml | 4 ++-- scalding/pom.xml| 4 ++-- scio/pom.xml| 4 ++-- shell/pom.xml | 4 ++-- spark/interpreter/pom.xml | 4 ++-- spark/pom.xml | 4 ++-- spark/scala-2.10/pom.xml| 4 ++-- spark/scala-2.11/pom.xml| 4 ++-- spark/spark-dependencies/pom.xml| 4 ++-- spark/spark-scala-parent/pom.xml| 4 ++-- spark/spark-shims/pom.xml | 4 ++-- spark/spark1-shims/pom.xml | 4 ++-- spark/spark2-shims/pom.xml | 4 ++-- zeppelin-display/pom.xml| 4 ++-- zeppelin-distribution/pom.xml | 2 +- zeppelin-examples/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-clock/pom.xml| 4 ++-- zeppelin-examples/zeppelin-example-horizontalbar/pom.xml| 4 ++-- zeppelin-examples/zeppelin-example-spell-echo/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-flowchart/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-markdown/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-translator/pom.xml | 4 ++-- zeppelin-integration/pom.xml| 4 ++-- zeppelin-interpreter/pom.xml| 4 ++-- zeppelin-jupyter/pom.xml| 4 ++-- zeppelin-server/pom.xml | 4 ++-- zeppelin-web/pom.xml| 4 ++-- zeppelin-zengine/pom.xml| 4 ++-- 52 files changed, 102 insertions(+), 102 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/41d9e0ac/alluxio/pom.xml -- diff --git a/alluxio/pom.xml b/alluxio/pom.xml index acef494..b6a4876 100644 --- a/alluxio/pom.xml +++ b/alluxio/pom.xml @@ -22,14 +22,14 @@ interpreter-parent org.apache.zeppelin -0.8.1-SNAPSHOT +0.8.0 ../interpreter-parent org.apache.zeppelin zeppelin-alluxio jar -0.8.1-SNAPSHOT +0.8.0 Zeppelin:
[2/2] zeppelin git commit: Preparing development version 0.8.1-SNAPSHOT
Preparing development version 0.8.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/d7918ab8 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/d7918ab8 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/d7918ab8 Branch: refs/heads/branch-0.8 Commit: d7918ab8d15a062fae51107ad954c9b7c6bf13c2 Parents: 41d9e0a Author: Jeff Zhang Authored: Tue May 29 21:37:00 2018 +0800 Committer: Jeff Zhang Committed: Tue May 29 21:37:00 2018 +0800 -- alluxio/pom.xml | 4 ++-- angular/pom.xml | 4 ++-- beam/pom.xml| 4 ++-- bigquery/pom.xml| 4 ++-- cassandra/pom.xml | 4 ++-- elasticsearch/pom.xml | 4 ++-- file/pom.xml| 4 ++-- flink/pom.xml | 4 ++-- geode/pom.xml | 4 ++-- groovy/pom.xml | 4 ++-- hbase/pom.xml | 4 ++-- helium-dev/pom.xml | 4 ++-- ignite/pom.xml | 4 ++-- interpreter-parent/pom.xml | 4 ++-- jdbc/pom.xml| 4 ++-- kylin/pom.xml | 4 ++-- lens/pom.xml| 4 ++-- livy/pom.xml| 4 ++-- markdown/pom.xml| 4 ++-- neo4j/pom.xml | 4 ++-- pig/pom.xml | 4 ++-- pom.xml | 2 +- python/pom.xml | 4 ++-- r/pom.xml | 4 ++-- sap/pom.xml | 4 ++-- scalding/pom.xml| 4 ++-- scio/pom.xml| 4 ++-- shell/pom.xml | 4 ++-- spark/interpreter/pom.xml | 4 ++-- spark/pom.xml | 4 ++-- spark/scala-2.10/pom.xml| 4 ++-- spark/scala-2.11/pom.xml| 4 ++-- spark/spark-dependencies/pom.xml| 4 ++-- spark/spark-scala-parent/pom.xml| 4 ++-- spark/spark-shims/pom.xml | 4 ++-- spark/spark1-shims/pom.xml | 4 ++-- spark/spark2-shims/pom.xml | 4 ++-- zeppelin-display/pom.xml| 4 ++-- zeppelin-distribution/pom.xml | 2 +- zeppelin-examples/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-clock/pom.xml| 4 ++-- zeppelin-examples/zeppelin-example-horizontalbar/pom.xml| 4 ++-- zeppelin-examples/zeppelin-example-spell-echo/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-flowchart/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-markdown/pom.xml | 4 ++-- zeppelin-examples/zeppelin-example-spell-translator/pom.xml | 4 ++-- zeppelin-integration/pom.xml| 4 ++-- zeppelin-interpreter/pom.xml| 4 ++-- zeppelin-jupyter/pom.xml| 4 ++-- zeppelin-server/pom.xml | 4 ++-- zeppelin-web/pom.xml| 4 ++-- zeppelin-zengine/pom.xml| 4 ++-- 52 files changed, 102 insertions(+), 102 deletions(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/d7918ab8/alluxio/pom.xml -- diff --git a/alluxio/pom.xml b/alluxio/pom.xml index b6a4876..acef494 100644 --- a/alluxio/pom.xml +++ b/alluxio/pom.xml @@ -22,14 +22,14 @@ interpreter-parent org.apache.zeppelin -0.8.0 +0.8.1-SNAPSHOT ../interpreter-parent org.apache.zeppelin zeppelin-alluxio jar -0.8.0 +0.8.1-SNAPSHOT Zeppelin: Alluxio interpreter
[zeppelin] Git Push Summary
Repository: zeppelin Updated Tags: refs/tags/v0.8.0-rc4 [created] 41d9e0ac3
zeppelin git commit: [HOTFIX] change string true to boolean true
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 306b0f23d -> 74ffc2134
[HOTFIX] change string true to boolean true
### What is this PR for?
Trivial fix
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
*
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang
Closes #2992 from zjffdu/hotfix_true and squashes the following commits:
cb0f956f1 [Jeff Zhang] [HOTFIX] change string true to boolean true
(cherry picked from commit 0c29a95bb653181feabec24659c05d91a027fcfe)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/74ffc213
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/74ffc213
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/74ffc213
Branch: refs/heads/branch-0.8
Commit: 74ffc213402f3b6a94fa91a55951a251c79f61d3
Parents: 306b0f2
Author: Jeff Zhang
Authored: Tue May 29 21:33:56 2018 +0800
Committer: Jeff Zhang
Committed: Tue May 29 21:35:37 2018 +0800
--
spark/interpreter/src/main/resources/interpreter-setting.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/74ffc213/spark/interpreter/src/main/resources/interpreter-setting.json
--
diff --git a/spark/interpreter/src/main/resources/interpreter-setting.json
b/spark/interpreter/src/main/resources/interpreter-setting.json
index c4bd442..8cd82bb 100644
--- a/spark/interpreter/src/main/resources/interpreter-setting.json
+++ b/spark/interpreter/src/main/resources/interpreter-setting.json
@@ -78,7 +78,7 @@
"zeppelin.spark.useNew": {
"envName": null,
"propertyName": "zeppelin.spark.useNew",
-"defaultValue": "true",
+"defaultValue": true,
"description": "Whether use new spark interpreter implementation",
"type": "checkbox"
}
zeppelin git commit: [HOTFIX] change string true to boolean true
Repository: zeppelin
Updated Branches:
refs/heads/master b68d36f08 -> 0c29a95bb
[HOTFIX] change string true to boolean true
### What is this PR for?
Trivial fix
### What type of PR is it?
[Hot Fix]
### Todos
* [ ] - Task
### What is the Jira issue?
*
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? NO
Author: Jeff Zhang
Closes #2992 from zjffdu/hotfix_true and squashes the following commits:
cb0f956f1 [Jeff Zhang] [HOTFIX] change string true to boolean true
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/0c29a95b
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/0c29a95b
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/0c29a95b
Branch: refs/heads/master
Commit: 0c29a95bb653181feabec24659c05d91a027fcfe
Parents: b68d36f
Author: Jeff Zhang
Authored: Tue May 29 21:33:56 2018 +0800
Committer: Jeff Zhang
Committed: Tue May 29 21:35:27 2018 +0800
--
spark/interpreter/src/main/resources/interpreter-setting.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/0c29a95b/spark/interpreter/src/main/resources/interpreter-setting.json
--
diff --git a/spark/interpreter/src/main/resources/interpreter-setting.json
b/spark/interpreter/src/main/resources/interpreter-setting.json
index c4bd442..8cd82bb 100644
--- a/spark/interpreter/src/main/resources/interpreter-setting.json
+++ b/spark/interpreter/src/main/resources/interpreter-setting.json
@@ -78,7 +78,7 @@
"zeppelin.spark.useNew": {
"envName": null,
"propertyName": "zeppelin.spark.useNew",
-"defaultValue": "true",
+"defaultValue": true,
"description": "Whether use new spark interpreter implementation",
"type": "checkbox"
}
zeppelin git commit: ZEPPELIN-3508. Enable new spark interpreter in 0.8.0
Repository: zeppelin
Updated Branches:
refs/heads/branch-0.8 2b1951495 -> 306b0f23d
ZEPPELIN-3508. Enable new spark interpreter in 0.8.0
### What is this PR for?
I'd like to enable it in 0.8 so that we can get more feedback from users. And
user can still use the old implementing via setting `zeppelin.spark.useNew` to
`false`
### What type of PR is it?
[Improvement]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-3508
### How should this be tested?
* CI pass
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
Author: Jeff Zhang
Closes #2989 from zjffdu/ZEPPELIN-3508 and squashes the following commits:
25c900b9f [Jeff Zhang] ZEPPELIN-3508. Enable new spark interpreter in 0.8.0
(cherry picked from commit ffa4ee31a2d45a0f8e4256cd3c285357c42410e1)
Signed-off-by: Jeff Zhang
Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/306b0f23
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/306b0f23
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/306b0f23
Branch: refs/heads/branch-0.8
Commit: 306b0f23dee217ce4d6f09abc63847e34974d844
Parents: 2b19514
Author: Jeff Zhang
Authored: Tue May 29 12:55:02 2018 +0800
Committer: Jeff Zhang
Committed: Tue May 29 21:32:28 2018 +0800
--
docs/interpreter/spark.md | 3 +--
spark/interpreter/src/main/resources/interpreter-setting.json | 2 +-
2 files changed, 2 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/306b0f23/docs/interpreter/spark.md
--
diff --git a/docs/interpreter/spark.md b/docs/interpreter/spark.md
index 85b2981..51b7c9e 100644
--- a/docs/interpreter/spark.md
+++ b/docs/interpreter/spark.md
@@ -205,8 +205,7 @@ You can either specify them in `zeppelin-env.sh`, or in
interpreter setting page
in interpreter setting page means you can use multiple versions of `spark` &
`hadoop` in one zeppelin instance.
### 4. New Version of SparkInterpreter
-There's one new version of SparkInterpreter starting with better spark support
and code completion from Zeppelin 0.8.0, by default we still use the old
version of SparkInterpreter.
-If you want to use the new one, you can configure `zeppelin.spark.useNew` as
`true` in its interpreter setting.
+There's one new version of SparkInterpreter with better spark support and code
completion starting from Zeppelin 0.8.0. We enable it by default, but user can
still use the old version of SparkInterpreter by setting
`zeppelin.spark.useNew` as `false` in its interpreter setting.
## SparkContext, SQLContext, SparkSession, ZeppelinContext
SparkContext, SQLContext and ZeppelinContext are automatically created and
exposed as variable names `sc`, `sqlContext` and `z`, respectively, in Scala,
Python and R environments.
http://git-wip-us.apache.org/repos/asf/zeppelin/blob/306b0f23/spark/interpreter/src/main/resources/interpreter-setting.json
--
diff --git a/spark/interpreter/src/main/resources/interpreter-setting.json
b/spark/interpreter/src/main/resources/interpreter-setting.json
index 30ae737..c4bd442 100644
--- a/spark/interpreter/src/main/resources/interpreter-setting.json
+++ b/spark/interpreter/src/main/resources/interpreter-setting.json
@@ -78,7 +78,7 @@
"zeppelin.spark.useNew": {
"envName": null,
"propertyName": "zeppelin.spark.useNew",
-"defaultValue": "false",
+"defaultValue": "true",
"description": "Whether use new spark interpreter implementation",
"type": "checkbox"
}
[zeppelin] Git Push Summary [forced push!] [Forced Update!]
Repository: zeppelin Updated Branches: refs/heads/branch-0.8 8a6133b5b -> 2b1951495 (forced update)
[zeppelin] Git Push Summary
Repository: zeppelin Updated Tags: refs/tags/v0.8.0-rc4 [deleted] 5fa23f651
svn commit: r27146 - /dev/zeppelin/zeppelin-0.8.0-rc4/
Author: zjffdu Date: Tue May 29 10:00:07 2018 New Revision: 27146 Log: Upload Zeppelin 0.8.0 RC4 Added: dev/zeppelin/zeppelin-0.8.0-rc4/ dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz (with props) dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.asc dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.sha512 dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz (with props) dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.asc dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.sha512 dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0.tgz (with props) dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0.tgz.asc dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0.tgz.sha512 Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz == Binary file - no diff available. Propchange: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz -- svn:mime-type = application/octet-stream Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.asc == --- dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.asc (added) +++ dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.asc Tue May 29 10:00:07 2018 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v2 + +iQIcBAABCAAGBQJbDOwTAAoJEAkPvhTZsX0fiCsP/jjk97qBlb0XE9v6YsTF7VPM +MYaI/m53W+tOuKZdv1ZB8nBX+RoxOnjHiNM33d9+ZurDyYk24xmSypim+yn5Ko/7 +iP8/x3Mn+yR0eqlfDQqb0MJp4ky2844VJiTr2wSnrztUcc5N5tzMNNEsbGQIEmNY +xJebrqbaLkY6GXB3DwQDEwgAou7XIEbukIgPhuR7qkKtXGxTqSUWlEJQSmK+xawL +ccxFmZZp6hQHvHJmhaWcy3u/kzQWMJPmgVQ/l9Q/uzbNWP5w6pXhXw2yAInpj2kF +ehMslfBfyKIDeYOvWnBXaxsCXXxLM9Y5F6sRU+Mnx9ALtSns1RdqbLah/xEAaKX6 +8xeZLrkSX4TbvjBysFiL/ywCfDwxPzHN0BM0+fINyrDbzbDfVB8p69ibvRSdWQhN +hN6WMsPZ03s6/nnrirveMS16FbIdulPaRj0VhqTfmDodipamWlTGX9ShImgDJPZp +MI7/GXgQQlSePF7MDFPUg6G0yFPWpqLu+Gc48rJUM1o2Ho9dv5U8Gy6+aZGWg/EB +ROeRV2ppgvK0iAHWso6nwJTci/DCCzZFKFiW4yyb6bQoQpbYzUxxCt1Q+tPyEddf +O5Rm5BBuJ55G98dwis57CfeyecJqGxD580V64zZcm42OPz6YVGCS+nJL0LPwsEUy +PVJ83kgGrTu8irTkAE/P +=OAgh +-END PGP SIGNATURE- Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.sha512 == --- dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.sha512 (added) +++ dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-all.tgz.sha512 Tue May 29 10:00:07 2018 @@ -0,0 +1 @@ +6e70904357bcb2a2faff12d93a65fe3163088415aad41749b34fa3c4ecdef5f9e4f3b93db345bd4ca26e10c51a36dc0577a60ffa0245065da73047ec5ce579bf zeppelin-0.8.0-bin-all.tgz Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz == Binary file - no diff available. Propchange: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz -- svn:mime-type = application/octet-stream Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.asc == --- dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.asc (added) +++ dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.asc Tue May 29 10:00:07 2018 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- +Version: GnuPG v2 + +iQIcBAABCAAGBQJbDO6IAAoJEAkPvhTZsX0fRUgQAIHVLfdAYLSRUEM8gCJqF9uu +3xkRNKBAZJv1y71C6HMt8efuDjCWUQygwnUgdCE696/thxDDb7MZe6CMVPuu5wpy +yg33zQWtstHjvxSoBUcMWb7K3cPuHxwJKNpaW3ZtS+szWqXBerkZqw8Bay0/ihnj +sBhJymgFyaulkOQGji51HEw8dzwjT4842I03i+6W/OhfA48sGNsQ+fYW1Xdbpu15 +A9oyMY2WIVYXM94tJv2O6qUBUCN2/1ZZbxjsTUz7fdGXOgH7rQudT0l1cMtM9W+i +Xssg6fJDu7R64Cp5F4FgNI3MjTWKgXTkWzJ+/qI7DPK5YExYT70UGXekzF6pNNVh +CHHPUYJW9LIFlVCdxN4hQfNxeaRKvnYn3VbqXA64cfTTkUcfavw2NRo+e1f3H91I +WRFxi3QIEeThpng3/S7N1KsLH7xon8jNiUykKrKFLqzdoZXjZffAvCc2wy0UrZDv +GUM9sKOTWC24/n644C/wOAqkhS4qUXl1G23p2n2UFlgOiqgpTpj7HGhHAtxZygmj +fXELXbIEuHiUXwqwy9JWumTUkgAp7dRfhvfR9IfHvJdHh9Drm4iNCCDROHXobFOX +PyCUzk4rwPuwhN4UABPiQtosJvp+59AQ/BF481f2qKaOMIGohPyqMGCf5F2rux2f +8ThpoyuvsPkbQ/nz3Qkb +=N8kX +-END PGP SIGNATURE- Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.sha512 == --- dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.sha512 (added) +++ dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0-bin-netinst.tgz.sha512 Tue May 29 10:00:07 2018 @@ -0,0 +1 @@ +70de2d2f1bed5f86ea3a12315371517eee6fa2117b7e9045fcb5ad35a61e32e0c8e93438b259d06b8cbb7f5985b54e073cf9111f891236cbe3f031836e55c32a zeppelin-0.8.0-bin-netinst.tgz Added: dev/zeppelin/zeppelin-0.8.0-rc4/zeppelin-0.8.0.tgz
zeppelin git commit: [HOTFIX] Don't copy dependencies to target/lib folder
Repository: zeppelin Updated Branches: refs/heads/master ffa4ee31a -> b68d36f08 [HOTFIX] Don't copy dependencies to target/lib folder ### What is this PR for? It is not necessary to copy dependencies jars to target/lib folder ### What type of PR is it? [Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring] ### Todos * [ ] - Task ### What is the Jira issue? * No jira created ### How should this be tested? * CI pass ### Screenshots (if appropriate) ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Jeff Zhang Closes #2990 from zjffdu/HOTFIX_pom and squashes the following commits: 224a3d77d [Jeff Zhang] [HOTFIX] Don't copy dependencies to target/lib folder Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/b68d36f0 Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/b68d36f0 Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/b68d36f0 Branch: refs/heads/master Commit: b68d36f08546b4f7cc223706f74ec3a09e68e419 Parents: ffa4ee3 Author: Jeff Zhang Authored: Tue May 29 14:02:11 2018 +0800 Committer: Jeff Zhang Committed: Tue May 29 15:39:13 2018 +0800 -- interpreter-parent/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/zeppelin/blob/b68d36f0/interpreter-parent/pom.xml -- diff --git a/interpreter-parent/pom.xml b/interpreter-parent/pom.xml index 3be64b3..25af441 100644 --- a/interpreter-parent/pom.xml +++ b/interpreter-parent/pom.xml @@ -78,7 +78,7 @@ maven-dependency-plugin - copy-interpreter-dependencies + copy-dependencies package copy-dependencies
