Github user mgaido91 commented on the issue:
https://github.com/apache/spark/pull/20856
@HyukjinKwon thanks for your great analysis. I agree with you that the
proposed fix is more a "workaround" than a real fix for the issue we have here.
The main problem here as you pointed out is that we have a bad (invalid?)
`FileSourceScanExec` on the executors. Probably this has never been an issue as
on the executors we accessed only some properties which were correctly
populated and we assumed that the other operation would have been performed
only on driver side.
I think the cleanest approach (not sure it is entirely feasible) would be
to choose one of the following option:
- check that all exec expression (in this case `FileSourceScanExec`) are
working properly both on driver and executor side;
- define which operation/attributes can be accessed on executor side too
and which only on driver side, document it and enforce it (if feasible).
What do you think?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
