[ https://issues.apache.org/jira/browse/SPARK-27598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stavros Kontopoulos updated SPARK-27598: ---------------------------------------- Priority: Critical (was: Blocker) > DStreams checkpointing does not work with Scala 2.12 > ---------------------------------------------------- > > Key: SPARK-27598 > URL: https://issues.apache.org/jira/browse/SPARK-27598 > Project: Spark > Issue Type: Bug > Components: DStreams > Affects Versions: 2.4.0, 2.4.1, 2.4.2, 3.0.0 > Reporter: Stavros Kontopoulos > Priority: Critical > > When I restarted a stream with checkpointing enabled I got this: > {quote}19/04/29 22:45:06 WARN CheckpointReader: Error reading checkpoint from > file > [file:/tmp/checkpoint/checkpoint-1556566950000.bk|file:///tmp/checkpoint/checkpoint-1556566950000.bk] > java.io.IOException: java.lang.ClassCastException: cannot assign instance of > java.lang.invoke.SerializedLambda to field > org.apache.spark.streaming.dstream.FileInputDStream.filter of type > scala.Function1 in instance of > org.apache.spark.streaming.dstream.FileInputDStream > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1322) > at > org.apache.spark.streaming.dstream.FileInputDStream.readObject(FileInputDStream.scala:314) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > {quote} > It seems that the closure is stored in the Serialized format and cannot be > assigned back to a scala function1 > Details of how to reproduce it here: > [https://gist.github.com/skonto/87d5b2368b0bf7786d9dd992a710e4e6] > Maybe this is spark-shell specific and is not expected to work anyway. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org