Github user viirya commented on the issue:
https://github.com/apache/spark/pull/20856
@HyukjinKwon @cloud-fan Thanks for pinging me, sorry for replying late.
Yeah I looked at the final fixing at #21815, it looks good for a fixing at this
particular problem.
> It seems to me it would be better to always do codegen at driver side, to
avoid complex expression/plan operations at executor side.(not sure if it's
possible, cc ...).
I do agree that this sounds better. A major part of executor codegen is
unsafe codegen classes such as unsafe projection. Most of them if not all are
not serializable for now. In order to do codegen at driver side at all, we may
need to make them serializable. Is it worth doing this?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
