See: https://issues.apache.org/jira/browse/SPARK-23131 https://github.com/apache/spark/pull/20301#issuecomment-358473199
I expected a major Kryo upgrade to be problematic, but it worked fine. It picks up a number of fixes: https://github.com/EsotericSoftware/kryo/releases/tag/kryo-parent-4.0.0 It might be good for Spark 2.4. Its serialized format isn't entirely compatible though. I'm trying to recall whether this is a problem in practice. We don't guarantee wire compatibility across mismatched Spark versions, right? But does the Kryo serialized form show up in any persistent stored form? I don't believe any normal output, even that of saveAsObjectFile, uses it. I'm wondering if I am not recalling why this would be a problem to update? Sean
