GitHub user Leemoonsoo opened a pull request:
https://github.com/apache/incubator-zeppelin/pull/703
[WIP] [ZEPPELIN-513] Dedicated interpreter session per notebook
### What is this PR for?
Currently, all notebooks that binded to the same interpreter setting shares
the same interpreter instance.
For example, let's say there're `spark-production` interpreter setting and
all notebooks are using it.
Then all notebooks share a single instance of SparkInterpreter, and that
result all the variables are being shared across the notebook.
But sometimes isolated name space for variables per notebook is preferred.
In this case we can create per notebook interpreter instance.
This PR provides generalized facility to provides per notebook interpreter
instance, not only for SparkInterpreter but for any other existing interpreter
without modification.
**Design**
It provides toggle switch that user turn on / turn off this mode per
interpreter setting.
When per notebook interpreter is turned on, it still creates a single
remote interpreter process per interpreter setting. But inside of the process,
each notebook will have it's own interpreter instance and the instance is
terminated automatically on removal of notebook or unbinding note from
interpreter setting.
**SparkInterpreter**
Because of the design keeps single remote interpreter process and creates
multiple interpreter instance inside of it, SparkInterpreter become a problem.
While multiple SparkContext can not be created on a single process. So this PR
try to, create per notebook scala compiler instance, but share single
SparkContext instance. In this way, each notebook can run job concurrently by
leveraging fairscheduler of SparkContext without running multiple Spark
application (SparkContext).
And this approach is inspired by
https://github.com/piyush-mukati/incubator-zeppelin/tree/parallel_scheduler_support_spark,
this PR trying to generalize it for all other interpreters.
### What type of PR is it?
Feature
### Todos
* [x] - Per note interpreter session
* [x] - Scala compiler instance per note and share a single SparkContext
(SparkInterpreter)
* [ ] - Handle Spark class server correctly
### Is there a relevant Jira issue?
https://issues.apache.org/jira/browse/ZEPPELIN-513
### How should this be tested?
Check "Per note session" of your Spark interpreter setting.
And create two different notebook and check scala variables are not shared.
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? no
* Is there breaking changes for older versions? no
* Does this needs documentation? yes
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Leemoonsoo/incubator-zeppelin
notebook_interpreter_session
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-zeppelin/pull/703.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #703
----
commit 5787984def99a63a465ab7a9414875414fc9b817
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-01T08:34:43Z
Add option for per note session interpreter
commit ed1ab0d21367b7a299e122446cbe56f4b283d9f2
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-02T21:33:10Z
zeppelin-interpreter note session support
commit fc9cb3f4b5bb3411aef3796857a7a26fdda821bd
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-03T16:39:48Z
interpreter session aware interpreter factory
commit 366b6519436589bf39be1896c38febbac4004775
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-06T23:12:43Z
Don't destroy interpreter on note memove or change bindings when
interpreter.option.perNoteSession is false
commit 58e5f917a4a9f14f8821b5db0a45738d854b9855
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-07T01:12:14Z
Update spark interpreter
commit 1d1638c27e85ed6613f3d8184c5ee9bbc224daaf
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-07T01:42:51Z
Fix test
commit 6dd981a0366f55926905849f17a6bdd48568feac
Author: Lee moon soo <m...@apache.org>
Date: 2016-02-07T02:17:29Z
update interpreter module test
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
