[
https://issues.apache.org/jira/browse/LIVY-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931234#comment-16931234
]
Naman Mishra commented on LIVY-666:
-----------------------------------
Proposed approach:
Currently, a Livy session contains one "interpretergroup" which is a map of
(Kind -> Interpreter objects) . In order to have multiple interpreter groups in
a session, we can add another field that holds (intpGroupId -> intpGroup
object) mapping. Interpreter groups will be created and added to this map
lazily as and when a new scope is requested for execution. Effectively, a
session will transform like below:
!multiple_interpreter_groups.png|width=791,height=335!
As can be seen from the above figure, all interpreter groups share a common
SparkContext. Execute API (POST /statements) will accept an optional parameter
"interpreterGroup" in the request body that can be passed to denote the
interpreter group under which the code will be executed. Since the REPLs of one
interpreter group are isolated from REPLs of other interpreter groups, variable
declarations, imports and other code constructs are scoped to an interpreter
group within a Livy session.
CC [~jerryshao] [~tc0312] Can you guys please take a look at the proposed
feature and approach and provide any feedback/suggestions? I can go ahead with
implementation if I have a heads up.
> Support named interpreter groups
> --------------------------------
>
> Key: LIVY-666
> URL: https://issues.apache.org/jira/browse/LIVY-666
> Project: Livy
> Issue Type: New Feature
> Affects Versions: 0.7.0
> Reporter: Naman Mishra
> Priority: Major
> Attachments: multiple_interpreter_groups.png
>
>
> Currently, a session can contain only one interpreter group. In order to
> support use case of multiple repls with the same spark application multiple
> interpreters with different variable scoping (something similar to scoped
> interpreter mode in Zeppelin:
> [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode
>
> |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]),
> I propose to have "named interpreter groups", i.e., multiple interpreter
> groups in a session all sharing a spark context. The interpreter group can be
> specified on which the execution is supposed to happen in the execution API.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
