i found jira that seems related:
https://issues.apache.org/jira/browse/SPARK-25047
On Fri, Mar 29, 2019 at 4:01 PM Koert Kuipers <ko...@tresata.com> wrote:
> hi all,
> we are switching from scala 2.11 to 2.12 with a spark 2.4.1 release
> candidate and so far this has been going pretty smoothly.
>
> however we do see some new serialization errors related to Function1,
> Function2, etc.
>
> they look like this:
> ClassCastException: cannot assign instance of
> java.lang.invoke.SerializedLambda to field MyCaseClass.f of type
> scala.Function1 in instance of MyCaseClass
> at
> java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2133)
>
> these are often simple case classes with inside a val for the function.
> like this:
> case class MyCaseClass[X, Y](...) {
> val f: Function1[X, Y] = ...
> }
>
> we had no problems with these in scala 2.11. it does not look like these
> classes have members that are not serializable, and neither do the
> functions close over anything troublesome. since we get this for some
> classes but not for others i am not entirely sure what to make of it. we
> can work around the issue by changing the val f to a def, like this:
> case class MyCaseClass[X, Y](...) {
> def f: Function1[X, Y] = ...
> }
>
> any idea what is causing this?
> thanks!
> koert
>
>
>
