Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21157 I don't like the hack too but the complete removal just basically means we are going to drop namedtuple supports in RDD without, for example, any deprecation warnings. Spark is being super conservative and this's going to break compatibility. So, I was thinking we could do this for Spark 3.0. We already start to talk about this. This should probably be something we should discuss in the mailing list since it's a breaking change. One thing clear is that the complete removal should target 3.0.0. For now, yea, `cloudpickle` solves it but probably it's less performant. Logically, `cloudpickle` [fixed it](https://github.com/cloudpipe/cloudpickle/pull/113) (our cloudpickle copy is matched to [0.4.3](https://github.com/cloudpipe/cloudpickle/releases/tag/v0.4.3)) and we can take after the fix with the normal pickle side, aren't we?
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org