Osira Ben created SPARK-27137:
---------------------------------
Summary: Spark captured variable is null if the code is pasted via
:paste
Key: SPARK-27137
URL: https://issues.apache.org/jira/browse/SPARK-27137
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Osira Ben
If I execute this piece of code
{code:java}
val foo = "foo"
def f(arg: Any): Unit = {
Option(42).foreach(_ => java.util.Objects.requireNonNull(foo, "foo"))
}
sc.parallelize(Seq(1, 2), 2).foreach(f)
{code}
{{in spark2-shell via :paste it throws}}{{}}
{code:java}
scala> :paste
// Entering paste mode (ctrl-D to finish)
val foo = "foo"
def f(arg: Any): Unit = {
Option(42).foreach(_ => java.util.Objects.requireNonNull(foo, "foo"))
}
sc.parallelize(Seq(1, 2), 2).foreach(f)
// Exiting paste mode, now interpreting.
19/03/11 15:02:06 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0
(TID 2, hadoop.company.com, executor 1): java.lang.NullPointerException: foo
at java.util.Objects.requireNonNull(Objects.java:228)
{code}
However if I execute it pasting without :paste or via spark2-shell -i it
doesn't.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
