Kris Mok created SPARK-25138:
--------------------------------
Summary: Spark Shell should show the Scala prompt after
initialization is complete
Key: SPARK-25138
URL: https://issues.apache.org/jira/browse/SPARK-25138
Project: Spark
Issue Type: Bug
Components: Spark Shell
Affects Versions: 2.4.0
Reporter: Kris Mok
In previous Spark versions, the Spark Shell used to only show the Scala prompt
*after* Spark has initialized. i.e. when the user is able to enter code, the
Spark context, Spark session etc have all completed initialization, so {{sc}},
{{spark}} are all ready to use.
In the current Spark master branch (to become Spark 2.4.0), the Scala prompt
shows up immediately, while Spark itself is still in initialization in the
background. It's very easy for the user to feel as if the shell is ready and
start typing, only to find that Spark isn't ready yet, and Spark's
initialization logs get in the way of typing. This new behavior is rather
annoying from a usability's perspective.
A typical startup of the Spark Shell in current master:
{code:none}
$ bin/spark-shell
18/08/16 23:18:05 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0-SNAPSHOT
/_/
Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.range(1)Spark context Web UI available at http://localhost:4040
Spark context available as 'sc' (master = local[*], app id =
local-1534486692744).
Spark session available as 'spark'.
.show
+---+
| id|
+---+
| 0|
+---+
scala>
{code}
Could you see that it was running {{spark.range(1).show}} ?
In contrast, previous versions of Spark Shell would wait for Spark to fully
initialization:
{code:none}
$ bin/spark-shell
18/08/16 23:20:05 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use
setLogLevel(newLevel).
Spark context Web UI available at http://10.0.0.76:4040
Spark context available as 'sc' (master = local[*], app id =
local-1534486813159).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.3-SNAPSHOT
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_131)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.range(1).show
+---+
| id|
+---+
| 0|
+---+
scala>
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
