[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[
https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897536#comment-15897536
]
Luciano Resende commented on TOREE-390:
---
Are you seeing this issue because you are using local[*] spark, that gets
initiated when toree gets initiated? Would you have the same issue if you just
connect to an existing Spark Cluster ?
The problem with that is that the kernel would have sent the 'idle' status
telling the consumer he is ready to process requests, while he is not. And when
requests are received, then we get all sorts of strange issues such as sc not
defined, or some weird scala issues etc which was recently in Toree before
the following fix:
https://github.com/apache/incubator-toree/commit/6ef0c8cec02a6b622e204ab0271321e0f8a75d38
> Lazily start Spark sessions
> ---
>
> Key: TOREE-390
> URL: https://issues.apache.org/jira/browse/TOREE-390
> Project: TOREE
> Issue Type: Improvement
>Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is
> taken by starting a Spark session and waiting for containers. Lazily starting
> Spark sessions helps the notebook environment feel faster, even if the user
> is waiting on Spark to start up because the time waiting for Spark to start
> is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that
> can't be changed in a Spark context. It also enables the same startup code
> that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
> .config(...)
> .setAppName(...)
> .getOrCreate()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[
https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990490#comment-15990490
]
Ryan Blue commented on TOREE-390:
-
[~luciano resende], we aren't having the weird issues with requests any more,
since the commit you linked to and
[8f1775d|https://github.com/apache/incubator-toree/commit/8f1775d9dcdc56dd4f3a7c44abf0c36ae0417444].
The problem this fixes is the initial wait time is both startup for Toree and
Spark. This separates the two waits, making it clear which one is due to Spark
starting up and getting resources from YARN (in our deployment). This is better
for the user experience because people don't think Toree is slow, they
correctly attribute it to Spark. The secondary benefit is that users can
customize their Spark session before it starts by creating one in the first
cell, and Toree works just fine with that.
> Lazily start Spark sessions
> ---
>
> Key: TOREE-390
> URL: https://issues.apache.org/jira/browse/TOREE-390
> Project: TOREE
> Issue Type: Improvement
>Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is
> taken by starting a Spark session and waiting for containers. Lazily starting
> Spark sessions helps the notebook environment feel faster, even if the user
> is waiting on Spark to start up because the time waiting for Spark to start
> is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that
> can't be changed in a Spark context. It also enables the same startup code
> that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
> .config(...)
> .setAppName(...)
> .getOrCreate()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[
https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040785#comment-16040785
]
ASF GitHub Bot commented on TOREE-390:
--
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-toree/pull/109
> Lazily start Spark sessions
> ---
>
> Key: TOREE-390
> URL: https://issues.apache.org/jira/browse/TOREE-390
> Project: TOREE
> Issue Type: Improvement
>Reporter: Ryan Blue
>
> In our deployment, more than half of the startup time for a Toree notebook is
> taken by starting a Spark session and waiting for containers. Lazily starting
> Spark sessions helps the notebook environment feel faster, even if the user
> is waiting on Spark to start up because the time waiting for Spark to start
> is clearly Spark, not Toree, and is initiated by the user.
> Also, lazily starting a Spark session allows users to change settings that
> can't be changed in a Spark context. It also enables the same startup code
> that would be used in a spark-submit application:
> {code:lang=java}
> SparkSession.builder
> .config(...)
> .setAppName(...)
> .getOrCreate()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
