GitHub user jongyoul opened a pull request:
https://github.com/apache/zeppelin/pull/2329
[WIP][PoC] ZEPPELIN-2040 ClusterManager to support launching interpreter in
a cluster
### What is this PR for?
Launching interpreters into yarn cluster. This is PoC level now and has
many more steps described below. The main classes are `Client`,
`ClusterManager`, `RemoteInterpreterYarnProcess`
### What type of PR is it?
[Bug Fix | Improvement | Feature | Documentation | Hot Fix | Refactoring]
### Todos
* [ ] - Divide yarn dependencies with other module
* [ ] - Support yarn-cluster without setting SPARK_HOME
* [ ] - Remove unused files
* [ ] - TBD
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELI-2040
### How should this be tested?
1. Install hadoop in your local
https://dtflaneur.wordpress.com/2015/10/02/installing-hadoop-on-mac-osx-el-capitan/
1. add `zeppelin.cluster_manager=yarn` in your spark interpreter setting
1. test spark script
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? No
* Is there breaking changes for older versions? No
* Does this needs documentation? No
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jongyoul/zeppelin ZEPPELIN-2040
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/zeppelin/pull/2329.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2329
----
commit 4498cbe0bc46aa7f3e66ea3c1c5a38c750d1577c
Author: Jongyoul Lee <[email protected]>
Date: 2017-04-15T15:39:37Z
First step to implement from scratch
commit db753c4cd2253911c70935beb2d2e9bd46710e74
Author: Jongyoul Lee <[email protected]>
Date: 2017-04-23T21:47:31Z
Added license header for avoiding rat failed
commit 96b610760406521e82c8cbeeec2aea977324b5a4
Author: Jongyoul Lee <[email protected]>
Date: 2017-04-24T09:14:51Z
WIP
commit 0c3f898f53b7a4931993289019d29702c54bd696
Author: Jongyoul Lee <[email protected]>
Date: 2017-04-24T14:55:08Z
WIP
commit 060fb2742a32e8c2b211ad75f921bbdaf8322a3e
Author: Jongyoul Lee <[email protected]>
Date: 2017-04-30T04:34:12Z
WIP
commit 6117bef9a64bf6abf6aa2d61241d6bc4a66c7c01
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-01T08:57:40Z
remove zeppelin-cluster/yarn
commit 91a42b83005ec7bd6fde8b8ea6ac6c37a5ae021b
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-01T23:56:43Z
WIP
commit 0fbf81568581f57c5cb4b7e416931328f2d79e5e
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-03T16:10:52Z
POC in my local
commit 2bb895a136e5527e90725df563433ab60cb2b142
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-07T15:49:37Z
Fixed rat issue
commit c31d19d9f2e5fda2a6152ae8910b13274d77d9ce
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-07T16:23:33Z
Added license header
commit 8b6b1b85edd9ccaa8dd5f27559c590cd734b15c1
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-10T00:37:48Z
First version of yarn cluster manager
commit 09a8fe8a9b4d579fb7d0c56e77a00142f577021c
Author: Jongyoul Lee <[email protected]>
Date: 2017-05-10T03:01:18Z
Fixed style
Fixed rat issue
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
