Make sure you initialize a log4j Log object on the workers and not on the driver program. If you’re somehow referencing a logInfo method on the driver program, the Log object might not get sent across the network correctly (though you’d usually get some other error there, like NotSerializableException). Maybe try using println() at first.
As Andrew said, the log output will go into the stdout and stderr files in the work directory on your worker. You can also access those from the Spark cluster’s web UI (click on a worker there, then click on stdout / stderr). Matei On Apr 4, 2014, at 11:49 AM, John Salvatier <jsalvat...@gmail.com> wrote: > Is there a way to log exceptions inside a mapping function? logError and > logInfo seem to freeze things. > > > On Fri, Apr 4, 2014 at 11:02 AM, Matei Zaharia <matei.zaha...@gmail.com> > wrote: > Exceptions should be sent back to the driver program and logged there (with a > SparkException thrown if a task fails more than 4 times), but there were some > bugs before where this did not happen for non-Serializable exceptions. We > changed it to pass back the stack traces only (as text), which should always > work. I’d recommend trying a newer Spark version, 0.8 should be easy to > upgrade to from 0.7. > > Matei > > On Apr 4, 2014, at 10:40 AM, John Salvatier <jsalvat...@gmail.com> wrote: > > > I'm trying to get a clear idea about how exceptions are handled in Spark? > > Is there somewhere where I can read about this? I'm on spark .7 > > > > For some reason I was under the impression that such exceptions are > > swallowed and the value that produced them ignored but the exception is > > logged. However, right now we're seeing the task just re-tried over and > > over again in an infinite loop because there's a value that always > > generates an exception. > > > > John > >
