Exceptions should be sent back to the driver program and logged there (with a SparkException thrown if a task fails more than 4 times), but there were some bugs before where this did not happen for non-Serializable exceptions. We changed it to pass back the stack traces only (as text), which should always work. I’d recommend trying a newer Spark version, 0.8 should be easy to upgrade to from 0.7.
Matei On Apr 4, 2014, at 10:40 AM, John Salvatier <jsalvat...@gmail.com> wrote: > I'm trying to get a clear idea about how exceptions are handled in Spark? Is > there somewhere where I can read about this? I'm on spark .7 > > For some reason I was under the impression that such exceptions are swallowed > and the value that produced them ignored but the exception is logged. > However, right now we're seeing the task just re-tried over and over again in > an infinite loop because there's a value that always generates an exception. > > John
