[jira] [Comment Edited] (LIVY-664) Spark application still running when Livy session creating was rejected
[
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928495#comment-16928495
]
Oleksandr Shevchenko edited comment on LIVY-664 at 9/12/19 12:39 PM:
-
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet (application for duplicated session won't be submitted) but if
application is already in ACCEPTED or RUNNING state stop session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code:java}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before submitting the application, but I don't
see where we actually do it. So, I put off this bug fix since little busy now.
I can continue my work on this issue if you help me to find application
submission code (hope this is not a Python code, but I didn't find related
Scala/Java code for the first time).
Thanks!
was (Author: oshevchenko):
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code:java}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before submitting the application, but I don't
see where we actually do it. So, I put off this bug fix since little busy now.
I can continue my work on this issue if you help me to find application
submission code (hope this is not a Python code, but I didn't find related
Scala/Java code for the first time).
Thanks!
> Spark application still running when Livy session creating was rejected
>
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png,
> image-2019-09-08-20-39-18-569.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
> 2.1 Second session creating will be rejected since duplicated session name
> is not allowed.
> 2.2 Spark application will be submitted but Livy session won't be created
>
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (LIVY-664) Spark application still running when Livy session creating was rejected
[
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928495#comment-16928495
]
Oleksandr Shevchenko edited comment on LIVY-664 at 9/12/19 12:35 PM:
-
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code:java}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before submitting the application, but I don't
see where we actually do it. So, I put off this bug fix since little busy now.
I can continue my work on this issue if you help me to find application
submission code (hope this is not a Python code, but I didn't find related
Scala/Java code for the first time).
Thanks!
was (Author: oshevchenko):
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code:java}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before submitting the application, but I don't
see where we actually do it. So, I put off this bug fix since little busy now.
I can continue my work on this issue if you help me to find where an
application is submitted (hope this is not a Python code, but I didn't find
related Scala/Java code for the first time).
Thanks!
> Spark application still running when Livy session creating was rejected
>
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png,
> image-2019-09-08-20-39-18-569.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
> 2.1 Second session creating will be rejected since duplicated session name
> is not allowed.
> 2.2 Spark application will be submitted but Livy session won't be created
>
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (LIVY-664) Spark application still running when Livy session creating was rejected
[
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928495#comment-16928495
]
Oleksandr Shevchenko edited comment on LIVY-664 at 9/12/19 12:34 PM:
-
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code:java}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before submitting the application, but I don't
see where we actually do it. So, I put off this bug fix since little busy now.
I can continue my work on this issue if you help me to find where an
application is submitted (hope this is not a Python code, but I didn't find
related Scala/Java code for the first time).
Thanks!
was (Author: oshevchenko):
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before start application, but I don't see
where we actually do it. So, I put off this bug fix since little busy now. I
can continue my work on this issue if you help me to find where an application
is submitted (hope this is not a Python code, but I didn't find related
Scala/Java code for the first time).
Thanks!
> Spark application still running when Livy session creating was rejected
>
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png,
> image-2019-09-08-20-39-18-569.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
> 2.1 Second session creating will be rejected since duplicated session name
> is not allowed.
> 2.2 Spark application will be submitted but Livy session won't be created
>
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected
[
https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928495#comment-16928495
]
Oleksandr Shevchenko commented on LIVY-664:
---
Thanks for your comment [~yihengw]!
Don't see that duplicated session stops now, we just throw an exception
([https://github.com/apache/incubator-livy/blob/def6318c84f32a09f219065691857f11ca74e7cb/server/src/main/scala/org/apache/livy/sessions/SessionManager.scala#L99]).
Could you show where we stop this session?
I have tried to add `session.stop()` before throwing IllegalArgumentException
and it works fine in the case when YARN/Spark standalone application isn't
started yet but if application is already in ACCEPTED or RUNNING state stop
session doesn't work.
By design, in this case, the application should be killed here
([https://github.com/apache/incubator-livy/blob/80daadef02ae57b2a5487c6f92e0f7df558d4864/server/src/main/scala/org/apache/livy/server/interactive/InteractiveSession.scala#L474]),
I guess. But we have bad error handling in RSCClient.java and error will not
be propagated and also `app` in the following code is None since
`session.start()` didn't execute:
{code}
app.foreach {
warn(s"Failed to stop RSCDriver. Killing it...")
_.kill()
}
{code}
Better to check the session name before start application, but I don't see
where we actually do it. So, I put off this bug fix since little busy now. I
can continue my work on this issue if you help me to find where an application
is submitted (hope this is not a Python code, but I didn't find related
Scala/Java code for the first time).
Thanks!
> Spark application still running when Livy session creating was rejected
>
>
> Key: LIVY-664
> URL: https://issues.apache.org/jira/browse/LIVY-664
> Project: Livy
> Issue Type: Bug
>Reporter: Oleksandr Shevchenko
>Priority: Major
> Attachments: image-2019-09-08-20-38-50-195.png,
> image-2019-09-08-20-39-18-569.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Steps for reproduce:
> 1. Create a session with some name
> 2. Create a second session with the same name
> 2.1 Second session creating will be rejected since duplicated session name
> is not allowed.
> 2.2 Spark application will be submitted but Livy session won't be created
>
> Result: Spark application was submitted but Livy session is not created
> Expected result: Livy session creating rejected AND Spark application should
> be finished with failed or killed state or even should not be submitted.
> !image-2019-09-08-20-38-50-195.png!
> !image-2019-09-08-20-39-18-569.png!
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (LIVY-325) Refactoring Livy Session and Interpreter
[ https://issues.apache.org/jira/browse/LIVY-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928493#comment-16928493 ] Naman Mishra commented on LIVY-325: --- [~jerryshao] Created a separate Jira for now for named interpreter group feature since the scope of this Jira is bigger: https://issues.apache.org/jira/browse/LIVY-666 > Refactoring Livy Session and Interpreter > > > Key: LIVY-325 > URL: https://issues.apache.org/jira/browse/LIVY-325 > Project: Livy > Issue Type: New Feature > Components: REPL >Affects Versions: 0.4.0 >Reporter: Saisai Shao >Priority: Major > > Currently in Livy master code, Livy interpreter is bound with Livy session, > when we created a session, we should specify which interpreter we want, and > this interpreter will be created implicitly. This potentially has several > limitations: > 1. We cannot create a share session, when we choose one language, we have to > create a new session to use. But some notebooks like Zeppelin could use > python, scala, R to manipulate data under the same SparkContext. So in Livy > we should decouple interpreter with SC and support shared context between > different interpreters. > 2. Furthermore, we cannot create multiple same interpreters in one session. > For example in Zeppelin scope mode, it could create multiple scala > interpreters to share with one context, but unfortunately in current Livy we > could not support this. > So based on the problems we mentioned above, we mainly have three things: > 1. Decouple interpreters from Spark context, so that we could create multiple > interpreters under one context. > 2. Make sure multiple interpreters could be worked together. > 3. Change REST APIs to support multiple interpreters per session. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (LIVY-666) Support named interpreter groups
Naman Mishra created LIVY-666: - Summary: Support named interpreter groups Key: LIVY-666 URL: https://issues.apache.org/jira/browse/LIVY-666 Project: Livy Issue Type: New Feature Affects Versions: 0.7.0 Reporter: Naman Mishra Currently, a session can contain only one interpreter group. In order to support use case of multiple repls with the same spark application multiple interpreters with different variable scoping (something similar to scoped interpreter mode in Zeppelin: [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]), I propose to have "named interpreter groups", i.e., multiple interpreter groups in a session all sharing a spark context. The interpreter group can be specified on which the execution is supposed to happen in the execution API. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (LIVY-664) Spark application still running when Livy session creating was rejected
[ https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928486#comment-16928486 ] Yiheng Wang edited comment on LIVY-664 at 9/12/19 12:18 PM: In the code, it will stop the duplicated session when finding there're duplicated names. In my test, I find the stop session may fail. I guess it may due to some race condition. I'm looking into it. Another suggestion I think it's great. We should check the name duplication before submitting the application. I'll raise a patch to fix that. was (Author: yihengw): In the code, it will stop the duplicated session when finding there're duplicated names. In my test, I find the stop session may fail. I guess it may due to some race condition. I'm looking into it. Another suggestion I think it good. We should check the name duplication before submitting the application. I'll raise a patch to fix that. > Spark application still running when Livy session creating was rejected > > > Key: LIVY-664 > URL: https://issues.apache.org/jira/browse/LIVY-664 > Project: Livy > Issue Type: Bug >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: image-2019-09-08-20-38-50-195.png, > image-2019-09-08-20-39-18-569.png > > Time Spent: 10m > Remaining Estimate: 0h > > Steps for reproduce: > 1. Create a session with some name > 2. Create a second session with the same name > 2.1 Second session creating will be rejected since duplicated session name > is not allowed. > 2.2 Spark application will be submitted but Livy session won't be created > > Result: Spark application was submitted but Livy session is not created > Expected result: Livy session creating rejected AND Spark application should > be finished with failed or killed state or even should not be submitted. > !image-2019-09-08-20-38-50-195.png! > !image-2019-09-08-20-39-18-569.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-664) Spark application still running when Livy session creating was rejected
[ https://issues.apache.org/jira/browse/LIVY-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928486#comment-16928486 ] Yiheng Wang commented on LIVY-664: -- In the code, it will stop the duplicated session when finding there're duplicated names. In my test, I find the stop session may fail. I guess it may due to some race condition. I'm looking into it. Another suggestion I think it good. We should check the name duplication before submitting the application. I'll raise a patch to fix that. > Spark application still running when Livy session creating was rejected > > > Key: LIVY-664 > URL: https://issues.apache.org/jira/browse/LIVY-664 > Project: Livy > Issue Type: Bug >Reporter: Oleksandr Shevchenko >Priority: Major > Attachments: image-2019-09-08-20-38-50-195.png, > image-2019-09-08-20-39-18-569.png > > > Steps for reproduce: > 1. Create a session with some name > 2. Create a second session with the same name > 2.1 Second session creating will be rejected since duplicated session name > is not allowed. > 2.2 Spark application will be submitted but Livy session won't be created > > Result: Spark application was submitted but Livy session is not created > Expected result: Livy session creating rejected AND Spark application should > be finished with failed or killed state or even should not be submitted. > !image-2019-09-08-20-38-50-195.png! > !image-2019-09-08-20-39-18-569.png! -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (LIVY-325) Refactoring Livy Session and Interpreter
[ https://issues.apache.org/jira/browse/LIVY-325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928471#comment-16928471 ] Naman Mishra commented on LIVY-325: --- Is there a plan on supporting this? I can start a PR for supporting "named interpreter groups", i.e., multiple interpreter groups in a session sharing a spark context like the scoped mode in Zeppelin ( [https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode |https://zeppelin.apache.org/docs/0.8.0/usage/interpreter/interpreter_binding_mode.html#scoped-mode]) . The interpreter group can be specified on which the execution is supposed to happen. > Refactoring Livy Session and Interpreter > > > Key: LIVY-325 > URL: https://issues.apache.org/jira/browse/LIVY-325 > Project: Livy > Issue Type: New Feature > Components: REPL >Affects Versions: 0.4.0 >Reporter: Saisai Shao >Priority: Major > > Currently in Livy master code, Livy interpreter is bound with Livy session, > when we created a session, we should specify which interpreter we want, and > this interpreter will be created implicitly. This potentially has several > limitations: > 1. We cannot create a share session, when we choose one language, we have to > create a new session to use. But some notebooks like Zeppelin could use > python, scala, R to manipulate data under the same SparkContext. So in Livy > we should decouple interpreter with SC and support shared context between > different interpreters. > 2. Furthermore, we cannot create multiple same interpreters in one session. > For example in Zeppelin scope mode, it could create multiple scala > interpreters to share with one context, but unfortunately in current Livy we > could not support this. > So based on the problems we mentioned above, we mainly have three things: > 1. Decouple interpreters from Spark context, so that we could create multiple > interpreters under one context. > 2. Make sure multiple interpreters could be worked together. > 3. Change REST APIs to support multiple interpreters per session. -- This message was sent by Atlassian Jira (v8.3.2#803003)
