Github user willb commented on the pull request:
https://github.com/apache/spark/pull/143#issuecomment-37690238
Here's what I was thinking about that: I left the check in `DAGScheduler`
in place because preemptive checking is optional (and indeed not done
everywhere) and it seems like it is in general a pretty inexpensive thing to do
to make sure the error is reported where it's sensible at all (as far as I
could tell, `clean` may be called multiple times per closure in any case). I
absolutely agree that it would be ideal to do serializability checking (as well
as closure cleaning) only once per closure, though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
