[jira] [Commented] (LIVY-395) Run statements in parallel in a session

[Jeff Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210609#comment-16210609
 ] 

Jeff Zhang commented on LIVY-395:
-

It is not about run statements in parallelly. It it better to run multiple 
spark interpretes which share the same sparkcontext in one session. 

> Run statements in parallel in a session
> ---
>
> Key: LIVY-395
> URL: https://issues.apache.org/jira/browse/LIVY-395
> Project: Livy
>  Issue Type: Bug
>  Components: API
>Affects Versions: 0.3
>Reporter: kaushik ranjan
>
> Is there an option in Livy, similar to spark JobServer where we can run 
> statements in an interactive session in parallel?
> Apache JobServer has a option to specify the number of requests which we want 
> to run in parallel. Do we have something similar in Livy?
> Microsoft Azure HDI support team say - we can run as many statements in a 
> session in parallel as are the number of cores in a headnode in the HDI 
> cluster. But when we tested - it doesn't.
> Right now we see bad utilization of resources within a spark session 
> (following interactive session model). Right now we have session per user 
> model. I want to use the session for multiple users, but Livy's model of 
> running only one statement at a time, is acting as a blocker for allowing 
> multiple users to use the same session to run requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-395) Run statements in parallel in a session

[Hong Shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210482#comment-16210482
 ] 

Hong Shen commented on LIVY-395:


We also need this feature in our cluster, think about that, if a app have to 
take a long time and large resource to init,  after init, we have a lot of 
statement to run, it sounds good if we can run statements in parallel.


> Run statements in parallel in a session
> ---
>
> Key: LIVY-395
> URL: https://issues.apache.org/jira/browse/LIVY-395
> Project: Livy
>  Issue Type: Bug
>  Components: API
>Affects Versions: 0.3
>Reporter: kaushik ranjan
>
> Is there an option in Livy, similar to spark JobServer where we can run 
> statements in an interactive session in parallel?
> Apache JobServer has a option to specify the number of requests which we want 
> to run in parallel. Do we have something similar in Livy?
> Microsoft Azure HDI support team say - we can run as many statements in a 
> session in parallel as are the number of cores in a headnode in the HDI 
> cluster. But when we tested - it doesn't.
> Right now we see bad utilization of resources within a spark session 
> (following interactive session model). Right now we have session per user 
> model. I want to use the session for multiple users, but Livy's model of 
> running only one statement at a time, is acting as a blocker for allowing 
> multiple users to use the same session to run requests.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209886#comment-16209886
 ] 

ASF GitHub Bot commented on LIVY-411:
-

Github user ajbozarth commented on the issue:

https://github.com/apache/incubator-livy/pull/56
  
On the code side this LGTM, but I didn't test out the functionality


> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209398#comment-16209398
 ] 

ASF GitHub Bot commented on LIVY-411:
-

GitHub user jerryshao reopened a pull request:

https://github.com/apache/incubator-livy/pull/56

[LIVY-411][REPL] Fix session cannot start issue when python or r package is 
missing

## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/LIVY-411

In Livy 0.5.0, we supported multiple languages in one session, but it 
requires that all the packages should be available before session creation, 
such as R package and Python package, otherwise session will be failed to 
create. However, in some cases python or R package may be missing in Spark 
distro, this will make Livy fail to creation interactive session.

To fix this issue, we should not force such restriction on session 
creation, but delay the check until related interpreter is used. If such 
packaging is missing, we should make the related execution failure and return 
user the cause of issue, but don't affect other correctly started interpreters.


## How was this patch tested?

Existing test and manually verification on local cluster.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/incubator-livy LIVY-411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56






> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209397#comment-16209397
 ] 

ASF GitHub Bot commented on LIVY-411:
-

Github user jerryshao closed the pull request at:

https://github.com/apache/incubator-livy/pull/56


> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209324#comment-16209324
 ] 

ASF GitHub Bot commented on LIVY-411:
-

Github user jerryshao closed the pull request at:

https://github.com/apache/incubator-livy/pull/56


> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-409) Improve User Experience in livy-shell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209303#comment-16209303
 ] 

ASF GitHub Bot commented on LIVY-409:
-

Github user ericjperry commented on the issue:

https://github.com/apache/incubator-livy/pull/55
  
@ajbozarth @jerryshao Tests passed!


> Improve User Experience in livy-shell
> -
>
> Key: LIVY-409
> URL: https://issues.apache.org/jira/browse/LIVY-409
> Project: Livy
>  Issue Type: Improvement
>Reporter: Eric Perry
>Priority: Minor
>
> The livy-shell is useful in testing and evaluation of the use of Livy, but 
> has a few minor UX issues that could be fixed without many changes:
> # The use of httplib and a single connection can cause problems in 
> environments where network reliability is low.
> # The shell prompt does not include any contextual information, which may be 
> helpful when a user has multiple shells running.
> # There isn't an easy way to cancel the current command other than deleting 
> all the text on the prompt as SIGINT breaks out of the REPL and causes the 
> session to be deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209268#comment-16209268
 ] 

ASF GitHub Bot commented on LIVY-411:
-

GitHub user jerryshao reopened a pull request:

https://github.com/apache/incubator-livy/pull/56

[LIVY-411][REPL] Fix session cannot start issue when python or r package is 
missing

## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/LIVY-411

In Livy 0.5.0, we supported multiple languages in one session, but it 
requires that all the packages should be available before session creation, 
such as R package and Python package, otherwise session will be failed to 
create. However, in some cases python or R package may be missing in Spark 
distro, this will make Livy fail to creation interactive session.

To fix this issue, we should not force such restriction on session 
creation, but delay the check until related interpreter is used. If such 
packaging is missing, we should make the related execution failure and return 
user the cause of issue, but don't affect other correctly started interpreters.


## How was this patch tested?

Existing test and manually verification on local cluster.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/incubator-livy LIVY-411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56


commit 6f808d282757f3f4d6d38bdeb4dda59b167d9de5
Author: jerryshao 
Date:   2017-10-18T06:30:25Z

Fix session cannot start issue when python or r package is missing




> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209267#comment-16209267
 ] 

ASF GitHub Bot commented on LIVY-411:
-

Github user jerryshao closed the pull request at:

https://github.com/apache/incubator-livy/pull/56


> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-11) Enable HA support

[Hong Shen (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209248#comment-16209248
 ] 

Hong Shen commented on LIVY-11:
---

This feature is very necessary for livy, is there any update on the issue. 

> Enable HA support
> -
>
> Key: LIVY-11
> URL: https://issues.apache.org/jira/browse/LIVY-11
> Project: Livy
>  Issue Type: New Feature
>  Components: Core
>Affects Versions: 0.1
>Reporter: Romain Rigaux
>
> DB/HA: yes, just surviving a restart should be nice (I think Oozie allows 
> this, not sure about HS2, RM manager does not). HA like Sentry/HS2 with list 
> in ZooKeeper is nice



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-409) Improve User Experience in livy-shell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209244#comment-16209244
 ] 

ASF GitHub Bot commented on LIVY-409:
-

GitHub user ericjperry reopened a pull request:

https://github.com/apache/incubator-livy/pull/55

[LIVY-409] Livy shell UX improvements

I'm not sure how large of an audience the livy-shell has, but I've come to 
use it quite a bit in my testing/use of Livy and added a few small improvements 
that have made it a bit easier to use for me (and may help others):

* Replaced the use of httplib with requests to fix dropped connection 
issues (and a few other robustness issues).
* Added some context to the REPL prompt, by including the session `kind` 
and ID.
* Ignore `SIGINT` signal so that users can type `CTRL-C` to cancel out a 
command.

I know the use of `requests` is pretty ubiquitous but I included 
`livy-shell-requirements.txt` just in case 1) someone doesn't have it installed 
or 2) more dependencies are added down the road.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ericjperry/incubator-livy 
feature/livy-shell-improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/55.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #55


commit f45110c2f24d9b0d1e8645de09e7eda94b959487
Author: Eric Perry 
Date:   2017-10-05T15:48:37Z

Minor livy-shell UX improvements

commit da809a902423856b5c828d8c5a65ab4dcc33e95a
Author: Eric Perry 
Date:   2017-10-05T15:50:02Z

Add requirements file for livy-shell

commit 3b282afad13e3550387aba11850fbdc61ee01496
Author: Eric Perry 
Date:   2017-10-05T15:54:15Z

Remove commented code

commit 64a5abf423b9ae70eebaf04f6c78ef3f4c4862c2
Author: Eric Perry 
Date:   2017-10-05T18:05:46Z

Remove livy-shell-requirements.txt, check for requests at launch.

commit 494a0b32a3f61ff216a23a99e313f25cf8c59152
Author: Eric Perry 
Date:   2017-10-05T21:06:42Z

Use raise_for_status in favor of raising HTTPException

commit a8bf1f796d069df4a83d66432ac5a92aee4a6236
Author: Eric Perry 
Date:   2017-10-11T22:00:12Z

Fixed issue with commands being concatenated.

commit e09ff965f602913cfb82bc5ddb3432105511666a
Author: Eric Perry 
Date:   2017-10-13T13:32:11Z

Add parens to print and use message for python 3 compatibility

commit c78150e2859c0094ef9e16304f8b15123dab764e
Author: Eric Perry 
Date:   2017-10-13T13:51:07Z

Added CSRF token to requests




> Improve User Experience in livy-shell
> -
>
> Key: LIVY-409
> URL: https://issues.apache.org/jira/browse/LIVY-409
> Project: Livy
>  Issue Type: Improvement
>Reporter: Eric Perry
>Priority: Minor
>
> The livy-shell is useful in testing and evaluation of the use of Livy, but 
> has a few minor UX issues that could be fixed without many changes:
> # The use of httplib and a single connection can cause problems in 
> environments where network reliability is low.
> # The shell prompt does not include any contextual information, which may be 
> helpful when a user has multiple shells running.
> # There isn't an easy way to cancel the current command other than deleting 
> all the text on the prompt as SIGINT breaks out of the REPL and causes the 
> session to be deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-409) Improve User Experience in livy-shell

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209243#comment-16209243
 ] 

ASF GitHub Bot commented on LIVY-409:
-

Github user ericjperry closed the pull request at:

https://github.com/apache/incubator-livy/pull/55


> Improve User Experience in livy-shell
> -
>
> Key: LIVY-409
> URL: https://issues.apache.org/jira/browse/LIVY-409
> Project: Livy
>  Issue Type: Improvement
>Reporter: Eric Perry
>Priority: Minor
>
> The livy-shell is useful in testing and evaluation of the use of Livy, but 
> has a few minor UX issues that could be fixed without many changes:
> # The use of httplib and a single connection can cause problems in 
> environments where network reliability is low.
> # The shell prompt does not include any contextual information, which may be 
> helpful when a user has multiple shells running.
> # There isn't an easy way to cancel the current command other than deleting 
> all the text on the prompt as SIGINT breaks out of the REPL and causes the 
> session to be deleted.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (LIVY-412) CreateSession should be reject when too many child process "spark-submit" is running

[Hong Shen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LIVY-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hong Shen updated LIVY-412:
---
Summary: CreateSession should be reject when too many child process 
"spark-submit" is running  (was: CreateSession should be reject when there is 
too more child process "spark-submit" is running)

> CreateSession should be reject when too many child process "spark-submit" is 
> running
> 
>
> Key: LIVY-412
> URL: https://issues.apache.org/jira/browse/LIVY-412
> Project: Livy
>  Issue Type: Improvement
>  Components: Server
>Affects Versions: 0.4
>Reporter: Hong Shen
>
> In our cluster, livy server run with spark yarn cluster mode, when 
> createSession request is too frequently, livyServer will start too more 
> spark-submit child process, it will cause the machine oom. I think livy 
> server should reject the create session request when there is too more 
> spark-submit child process. I have fix it in our own cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (LIVY-412) CreateSession should be reject when there is too more child process "spark-submit" is running

[Hong Shen (JIRA)]

Hong Shen created LIVY-412:
--

 Summary: CreateSession should be reject when there is too more 
child process "spark-submit" is running
 Key: LIVY-412
 URL: https://issues.apache.org/jira/browse/LIVY-412
 Project: Livy
  Issue Type: Improvement
  Components: Server
Affects Versions: 0.4
Reporter: Hong Shen


In our cluster, livy server run with spark yarn cluster mode, when 
createSession request is too frequently, livyServer will start too more 
spark-submit child process, it will cause the machine oom. I think livy server 
should reject the create session request when there is too more spark-submit 
child process. I have fix it in our own cluster.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (LIVY-411) Session cannot be started when Python or R package is missing

[Saisai Shao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-411:
-
Reporter: Yesha Vora  (was: Saisai Shao)

> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Yesha Vora
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (LIVY-411) Session cannot be started when Python or R package is missing

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208872#comment-16208872
 ] 

ASF GitHub Bot commented on LIVY-411:
-

GitHub user jerryshao opened a pull request:

https://github.com/apache/incubator-livy/pull/56

[LIVY-411][REPL] Fix session cannot start issue when python or r package is 
missing

## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/LIVY-411

In Livy 0.5.0, we supported multiple languages in one session, but it 
requires that all the packages should be available before session creation, 
such as R package and Python package, otherwise session will be failed to 
create. However, in some cases python or R package may be missing in Spark 
distro, this will make Livy fail to creation interactive session.

To fix this issue, we should not force such restriction on session 
creation, but delay the check until related interpreter is used. If such 
packaging is missing, we should make the related execution failure and return 
user the cause of issue, but don't affect other correctly started interpreters.


## How was this patch tested?

Existing test and manually verification on local cluster.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jerryshao/incubator-livy LIVY-411

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-livy/pull/56.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #56


commit 6f808d282757f3f4d6d38bdeb4dda59b167d9de5
Author: jerryshao 
Date:   2017-10-18T06:30:25Z

Fix session cannot start issue when python or r package is missing




> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Saisai Shao
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (LIVY-411) Session cannot be started when Python or R package is missing

[Saisai Shao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LIVY-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saisai Shao updated LIVY-411:
-
Description: 
In Livy 0.5.0, we supported multiple languages in one session, but it requires 
that the packages should be available, such as R package and Python package, 
otherwise session will be failed to create. However, in some cases python or R 
package may be missing in Spark distro, this will make Livy fail to creation 
interactive session.

To fix this issue, we should not force such restriction on session creation, 
but delay the check until related interpreter is used. If such packaging is 
missing, we should make the related execution failure and return user the cause 
of issue, but don't affect other correctly started interpreters.

  was:
In Livy 0.5.0, we supported multiple languages in one session, but it requires 
one the packages be available, such as R package and Python package, otherwise 
session will be failed to create. But in some cases python or R package may be 
missing in Spark distro, this will make us fail to use Livy's interactive 
session.

To fix this issue, we should not force such restriction on session creation, 
but delay the check until related interpreter is used. If such packaging is 
missing, we should make the related execution failure and return user the cause 
of issue, but don't affect other correctly started interpreters.


> Session cannot be started when Python or R package is missing
> -
>
> Key: LIVY-411
> URL: https://issues.apache.org/jira/browse/LIVY-411
> Project: Livy
>  Issue Type: Bug
>  Components: REPL
>Affects Versions: 0.5.0
>Reporter: Saisai Shao
>Assignee: Saisai Shao
>
> In Livy 0.5.0, we supported multiple languages in one session, but it 
> requires that the packages should be available, such as R package and Python 
> package, otherwise session will be failed to create. However, in some cases 
> python or R package may be missing in Spark distro, this will make Livy fail 
> to creation interactive session.
> To fix this issue, we should not force such restriction on session creation, 
> but delay the check until related interpreter is used. If such packaging is 
> missing, we should make the related execution failure and return user the 
> cause of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (LIVY-411) Session cannot be started when Python or R package is missing

[Saisai Shao (JIRA)]

Saisai Shao created LIVY-411:


 Summary: Session cannot be started when Python or R package is 
missing
 Key: LIVY-411
 URL: https://issues.apache.org/jira/browse/LIVY-411
 Project: Livy
  Issue Type: Bug
  Components: REPL
Affects Versions: 0.5.0
Reporter: Saisai Shao
Assignee: Saisai Shao


In Livy 0.5.0, we supported multiple languages in one session, but it requires 
one the packages be available, such as R package and Python package, otherwise 
session will be failed to create. But in some cases python or R package may be 
missing in Spark distro, this will make us fail to use Livy's interactive 
session.

To fix this issue, we should not force such restriction on session creation, 
but delay the check until related interpreter is used. If such packaging is 
missing, we should make the related execution failure and return user the cause 
of issue, but don't affect other correctly started interpreters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)