[jira] [Created] (TOREE-390) Lazily start Spark sessions
Ryan Blue created TOREE-390:
---
Summary: Lazily start Spark sessions
Key: TOREE-390
URL: https://issues.apache.org/jira/browse/TOREE-390
Project: TOREE
Issue Type: Improvement
Reporter: Ryan Blue
In our deployment, more than half of the startup time for a Toree notebook is
taken by starting a Spark session and waiting for containers. Lazily starting
Spark sessions helps the notebook environment feel faster, even if the user is
waiting on Spark to start up because the time waiting for Spark to start is
clearly Spark, not Toree, and is initiated by the user.
Also, lazily starting a Spark session allows users to change settings that
can't be changed in a Spark context. It also enables the same startup code that
would be used in a spark-submit application:
{code:lang=java}
SparkSession.builder
.config(...)
.setAppName(...)
.getOrCreate()
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (TOREE-389) Scala interpreter hangs if Spark is referenced before it is bound
Ryan Blue created TOREE-389:
---
Summary: Scala interpreter hangs if Spark is referenced before it
is bound
Key: TOREE-389
URL: https://issues.apache.org/jira/browse/TOREE-389
Project: TOREE
Issue Type: Bug
Reporter: Ryan Blue
Some users run cells in a notebook before the kernel is finished starting. The
messages are queued and run after the KernelMessageRelay is enabled, but any
messages that reference {{spark}} or bound variables cause IMain to throw an
exception. That exception causes the interpreter to hang.
It seems odd that a missing variable reference causes IMain to throw an
exception, so there may be a threading issue, a race between binding {{spark}}
and the cell that references it.
The problem can be avoided by moving {{postInit}} tasks into init.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
