Repository: incubator-zeppelin Updated Branches: refs/heads/master 89c592489 -> 2ee234ac8
[ZEPPELIN-572] pyspark interpreter doesn't work on yarn-client ### What is this PR for? Set `spark.yarn.isPython` to be `true` to distribute pyspark libraries to workers when master is `yarn-client` ### What type of PR is it? Bug Fix ### Is there a relevant Jira issue? [ZEPPELIN-572](https://issues.apache.org/jira/browse/ZEPPELIN-572) ### How should this be tested? You need yarn cluster to test this PR. Simple way to test this PR would be running below code in paragraph and see if it throws error. ``` %pyspark print(sc.parallelize([1, 2]).count()) ``` And you should be able to see that `spark.yarn.isPython` is set to `true` in **Spark UI > Environment > Spark Properties** only when you set spark.master as `yarn-client`. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No Author: Mina Lee <[email protected]> Closes #605 from minahlee/ZEPPELIN-572 and squashes the following commits: 8c99de5 [Mina Lee] Set spark.yarn.isPython to be true to distribute needed pyspark libraries to workers when master is yarn-client Project: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/commit/2ee234ac Tree: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/tree/2ee234ac Diff: http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/diff/2ee234ac Branch: refs/heads/master Commit: 2ee234ac8e9248d671cba25349e3065c1bc326f3 Parents: 89c5924 Author: Mina Lee <[email protected]> Authored: Wed Jan 6 21:23:18 2016 -0800 Committer: Lee moon soo <[email protected]> Committed: Sun Jan 10 08:16:28 2016 -0800 ---------------------------------------------------------------------- .../main/java/org/apache/zeppelin/spark/SparkInterpreter.java | 4 ++++ 1 file changed, 4 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-zeppelin/blob/2ee234ac/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java ---------------------------------------------------------------------- diff --git a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java index 08eb0fa..2bf7a6b 100644 --- a/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java +++ b/spark/src/main/java/org/apache/zeppelin/spark/SparkInterpreter.java @@ -335,6 +335,10 @@ public class SparkInterpreter extends Interpreter { conf.set("spark.submit.pyArchives", Joiner.on(":").join(pythonLibs)); } + // Distributes needed libraries to workers. + if (getProperty("master").equals("yarn-client")) { + conf.set("spark.yarn.isPython", "true"); + } SparkContext sparkContext = new SparkContext(conf); return sparkContext;
