[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[ https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15897536#comment-15897536 ] Luciano Resende commented on TOREE-390: --- Are you seeing this issue because you are using local[*] spark, that gets initiated when toree gets initiated? Would you have the same issue if you just connect to an existing Spark Cluster ? The problem with that is that the kernel would have sent the 'idle' status telling the consumer he is ready to process requests, while he is not. And when requests are received, then we get all sorts of strange issues such as sc not defined, or some weird scala issues etc which was recently in Toree before the following fix: https://github.com/apache/incubator-toree/commit/6ef0c8cec02a6b622e204ab0271321e0f8a75d38 > Lazily start Spark sessions > --- > > Key: TOREE-390 > URL: https://issues.apache.org/jira/browse/TOREE-390 > Project: TOREE > Issue Type: Improvement >Reporter: Ryan Blue > > In our deployment, more than half of the startup time for a Toree notebook is > taken by starting a Spark session and waiting for containers. Lazily starting > Spark sessions helps the notebook environment feel faster, even if the user > is waiting on Spark to start up because the time waiting for Spark to start > is clearly Spark, not Toree, and is initiated by the user. > Also, lazily starting a Spark session allows users to change settings that > can't be changed in a Spark context. It also enables the same startup code > that would be used in a spark-submit application: > {code:lang=java} > SparkSession.builder > .config(...) > .setAppName(...) > .getOrCreate() > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[ https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990490#comment-15990490 ] Ryan Blue commented on TOREE-390: - [~luciano resende], we aren't having the weird issues with requests any more, since the commit you linked to and [8f1775d|https://github.com/apache/incubator-toree/commit/8f1775d9dcdc56dd4f3a7c44abf0c36ae0417444]. The problem this fixes is the initial wait time is both startup for Toree and Spark. This separates the two waits, making it clear which one is due to Spark starting up and getting resources from YARN (in our deployment). This is better for the user experience because people don't think Toree is slow, they correctly attribute it to Spark. The secondary benefit is that users can customize their Spark session before it starts by creating one in the first cell, and Toree works just fine with that. > Lazily start Spark sessions > --- > > Key: TOREE-390 > URL: https://issues.apache.org/jira/browse/TOREE-390 > Project: TOREE > Issue Type: Improvement >Reporter: Ryan Blue > > In our deployment, more than half of the startup time for a Toree notebook is > taken by starting a Spark session and waiting for containers. Lazily starting > Spark sessions helps the notebook environment feel faster, even if the user > is waiting on Spark to start up because the time waiting for Spark to start > is clearly Spark, not Toree, and is initiated by the user. > Also, lazily starting a Spark session allows users to change settings that > can't be changed in a Spark context. It also enables the same startup code > that would be used in a spark-submit application: > {code:lang=java} > SparkSession.builder > .config(...) > .setAppName(...) > .getOrCreate() > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (TOREE-390) Lazily start Spark sessions
[ https://issues.apache.org/jira/browse/TOREE-390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16040785#comment-16040785 ] ASF GitHub Bot commented on TOREE-390: -- Github user asfgit closed the pull request at: https://github.com/apache/incubator-toree/pull/109 > Lazily start Spark sessions > --- > > Key: TOREE-390 > URL: https://issues.apache.org/jira/browse/TOREE-390 > Project: TOREE > Issue Type: Improvement >Reporter: Ryan Blue > > In our deployment, more than half of the startup time for a Toree notebook is > taken by starting a Spark session and waiting for containers. Lazily starting > Spark sessions helps the notebook environment feel faster, even if the user > is waiting on Spark to start up because the time waiting for Spark to start > is clearly Spark, not Toree, and is initiated by the user. > Also, lazily starting a Spark session allows users to change settings that > can't be changed in a Spark context. It also enables the same startup code > that would be used in a spark-submit application: > {code:lang=java} > SparkSession.builder > .config(...) > .setAppName(...) > .getOrCreate() > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)