[
https://issues.apache.org/jira/browse/SPARK-27598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stavros Kontopoulos updated SPARK-27598:
----------------------------------------
Description:
When I restarted a stream with checkpointing enabled I got this:
{quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from
file
[file:/tmp/checkpoint/checkpoint-1556566950000.bk|file:///tmp/checkpoint/checkpoint-1556566950000.bk]
java.io.IOException: java.lang.ClassCastException: cannot assign instance of
java.lang.invoke.SerializedLambda to field
org.apache.spark.streaming.dstream.FileInputDStream.filter of type
scala.Function1 in instance of
org.apache.spark.streaming.dstream.FileInputDStream
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322)
at
org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}
It seems that the closure is stored in the Serialized format and cannot be
assigned back to a scala function1
Details of how to reproduce it here:
[https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6]
Maybe this is spark-shell specific.
was:
When I restarted a stream with checkpointing enabled I got this:
{quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from
file file:/tmp/checkpoint/checkpoint-1556566950000.bk
java.io.IOException: java.lang.ClassCastException: cannot assign instance of
java.lang.invoke.SerializedLambda to field
org.apache.spark.streaming.dstream.FileInputDStream.filter of type
scala.Function1 in instance of
org.apache.spark.streaming.dstream.FileInputDStream
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322)
at
org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}
It seems that the closure is stored in the Serialized format and cannot be
assigned back to a scala function1
Details of how to reproduce it here:
https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6
> DStreams checkpointing does not work with Scala 2.12
> ----------------------------------------------------
>
> Key: SPARK-27598
> URL: https://issues.apache.org/jira/browse/SPARK-27598
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0
> Reporter: Stavros Kontopoulos
> Priority: Blocker
>
> When I restarted a stream with checkpointing enabled I got this:
> {quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from
> file
> [file:/tmp/checkpoint/checkpoint-1556566950000.bk|file:///tmp/checkpoint/checkpoint-1556566950000.bk]
> java.io.IOException: java.lang.ClassCastException: cannot assign instance of
> java.lang.invoke.SerializedLambda to field
> org.apache.spark.streaming.dstream.FileInputDStream.filter of type
> scala.Function1 in instance of
> org.apache.spark.streaming.dstream.FileInputDStream
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322)
> at
> org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}
> It seems that the closure is stored in the Serialized format and cannot be
> assigned back to a scala function1
> Details of how to reproduce it here:
> [https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6]
> Maybe this is spark-shell specific.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
