[
https://issues.apache.org/jira/browse/SPARK-27598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16830745#comment-16830745
]
Stavros Kontopoulos edited comment on SPARK-27598 at 4/30/19 11:16 PM:
-----------------------------------------------------------------------
I will remove the language version from the title. This seems to be specific
to the shell.
With Spark 2.3.3 which I assume has 2.11 by default it fails with a different
error:
{quote}2019-05-01 02:12:00 WARN CheckpointReader:87 - Error reading checkpoint
from file
[file:/tmp/checkpoint/checkpoint-1556665870000.bk|file:///tmp/checkpoint/checkpoint-1556665870000.bk]
java.io.IOException: java.lang.ClassNotFoundException:
$line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$MapWithState$$anonfun$3
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1354)
at
org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:316)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
{quote}
I guess in this case its tries to de-serialize data from the checkpoint file
using classes from the Spark Shell that will not exist when the the new shell
is started like line16.
was (Author: skonto):
I will remove the language version from the title. This seems to be specific
to the shell.
With Spark 2.3.3 which I assume has 2.11 by default it fails with a different
error:
{quote}2019-05-01 02:12:00 WARN CheckpointReader:87 - Error reading checkpoint
from file
[file:/tmp/checkpoint/checkpoint-1556665870000.bk|file:///tmp/checkpoint/checkpoint-1556665870000.bk]
java.io.IOException: java.lang.ClassNotFoundException:
$line16.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$MapWithState$$anonfun$3
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1354)
at
org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:316)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1975)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
{quote}
I guess in this case its tries to deserialize data fromthe checkpoint file
using classes from the Spark Shell that will not exist when the the new shell
is started like line16.
> DStreams checkpointing does not work with the Spark Shell
> ---------------------------------------------------------
>
> Key: SPARK-27598
> URL: https://issues.apache.org/jira/browse/SPARK-27598
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0
> Reporter: Stavros Kontopoulos
> Priority: Major
>
> When I restarted a stream with checkpointing enabled I got this:
> {quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from
> file
> [file:/tmp/checkpoint/checkpoint-1556566950000.bk|file:///tmp/checkpoint/checkpoint-1556566950000.bk]
> java.io.IOException: java.lang.ClassCastException: cannot assign instance of
> java.lang.invoke.SerializedLambda to field
> org.apache.spark.streaming.dstream.FileInputDStream.filter of type
> scala.Function1 in instance of
> org.apache.spark.streaming.dstream.FileInputDStream
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322)
> at
> org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}
> It seems that the closure is stored in the Serialized format and cannot be
> assigned back to a scala function1
> Details of how to reproduce it here:
> [https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6]
> Maybe this is spark-shell specific and is not expected to work anyway, as I
> dont see this to be an issues with a normal jar.
> Note that with Spark 2.3.3 the error is different and this still does not
> work but with a different error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
