Local mode does serDe, so it should expose serialization problems.
On Wed, May 14, 2014 at 10:53 AM, Philip Ogren <philip.og...@oracle.com>wrote: > Have you actually found this to be true? I have found Spark local mode to > be quite good about blowing up if there is something non-serializable and > so my unit tests have been great for detecting this. I have never seen > something that worked in local mode that didn't work on the cluster because > of different serialization requirements between the two. Perhaps it is > different when using Kryo.... > > > > On 05/14/2014 04:34 AM, Andras Nemeth wrote: > >> E.g. if I accidentally use a closure which has something non-serializable >> in it, then my test will happily succeed in local mode but go down in >> flames on a real cluster. >> > >
