[GitHub] zeppelin pull request #2577: ZEPPELIN-2898. Support Yarn-Cluster for Spark I...
GitHub user zjffdu opened a pull request: https://github.com/apache/zeppelin/pull/2577 ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter ### What is this PR for? This is the first version for supporting yarn-cluster of `SparkInterpreter`. I just delegate all the function to `spark-submit` as yarn-cluster is natively supported by spark, we don't need to reinvent the wheel. But there's still improvement to be done in future, such as I put some spark specific login in `InterpreterSetting` which is not a good practise. I plan to improve it when I refactor the `Interpreter` class (ZEPPELIN-2685). Besides that, I also add `MiniHadoopCluster` & `MiniZeppelin` which help for the integration test of yarn-client & yarn-cluster mode, otherwise I have to manually verify yarn-client & yarn-cluster mode which would easily cause regression issue in future. ### What type of PR is it? [Feature] ### Todos * [ ] - Task ### What is the Jira issue? https://github.com/zjffdu/zeppelin/tree/ZEPPELIN-2898 ### How should this be tested? System test is added in `SparkInterpreterIT`. ### Questions: * Does the licenses files need update? No * Is there breaking changes for older versions? No * Does this needs documentation? No You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/zeppelin ZEPPELIN-2898 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/zeppelin/pull/2577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2577 commit 9ec46d931149eddc0a227cec00e1bee76180176d Author: Jeff Zhang Date: 2017-09-04T13:54:56Z ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpreter ---
[jira] [Created] (ZEPPELIN-2922) Allow removing last paragraph
Malay Majithia created ZEPPELIN-2922: Summary: Allow removing last paragraph Key: ZEPPELIN-2922 URL: https://issues.apache.org/jira/browse/ZEPPELIN-2922 Project: Zeppelin Issue Type: Improvement Reporter: Malay Majithia At present, we don't allow removing the last paragraph. Due to this sometimes user want to delete the last empty paragraph to make the notebook interface clean. This also requires handling the use case when there is only one paragraph: Fix: Don't allow removing the paragraph when there is only one paragraph in the notebook. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zeppelin issue #2577: ZEPPELIN-2898. Support Yarn-Cluster for Spark Interpre...
Github user zjffdu commented on the issue: https://github.com/apache/zeppelin/pull/2577 @Leemoonsoo @jongyoul Could you help review ? ---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Github user cloverhearts commented on the issue: https://github.com/apache/zeppelin/pull/2574 @zjffdu @felixcheung Thank you for your opinions :) I was left my comment. Could you check this one? ---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Github user cloverhearts commented on the issue: https://github.com/apache/zeppelin/pull/2574 @zjffdu Thank you for your opinion, I present scenarios for environment changes using conda. Python version and module names are arbitrary, so do not worry. What is important is change in every environment. 1. Basically Zeppelin's python python: 2.71 installed library: (none) use to zeppelin ``` %python import sys print(sys.version) print(sys.path) ``` result ``` python version 2.7.1 %ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME ``` ``` %python import myModule ``` result ``` no module myModule ``` 2. install conda and new python ``` %python.conda create --name python3 python=3.4 ``` ``` %python.conda activate python3 ``` ``` %python import sys print(sys.version) print(sys.path) ``` result ``` python version 3.4 %ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME:%CONDA_ENV%/python3/lib ``` ``` %python import myModule ``` result ``` no module myModule ``` try again ``` %python.conda install myModule ``` ``` import myModule ``` result ``` imported. ``` 3. restore python environment ``` %python.conda deactivate ``` check for recovered to the original Zeppelin environment. ``` %python import sys print(sys.version) print(sys.path) ``` result ``` python version 2.7.1 %ZEPPELIN_HOME%/interpreter/python/lib:%PYTHON_HOME ``` ``` import myModule ``` result ``` no module myModule ``` All three must be supported by conda. 1. Support to installing and using the python library 2. Support to change the python version 3. Support to configuring non-python environments old zeppelin does works for this part(0.71). but, the Python Interpreter has changed significantly since 0.71. I guess missing the environment part for this. I modified it. python env set code (https://github.com/apache/zeppelin/blob/master/python/src/main/java/org/apache/zeppelin/python/PythonInterpreter.java#L151) ---
[GitHub] zeppelin issue #2574: [ZEPPELIN-2921 : 0.7x] does not work conda environment...
Github user cloverhearts commented on the issue: https://github.com/apache/zeppelin/pull/2574 I think we are worried about `conda interpreter` and coupling for` conda env name`. However, the conda env name is more important to the python interpreter than the conda interpreter. Also, the python interpreter should not query the conda interpreter for its own state. I think this misunderstanding is due to the member name `conda env name`. If I have any additional problems, I will change this to `virtualEnvName`. ---
