[image: Inline image 1]
I'm not able to find the piece of code that you wrote, but you can use a
try...catch to catch your user specific exceptions and log it in the logs.
Something like this:
myRdd.map(x => try{ //something }catch{ case e:Exception =>
log.error("Whoops!! :" + e) })
Thanks
Best Regards
On Tue, Sep 1, 2015 at 1:22 AM, Wayne Song <wayne.e.s...@gmail.com> wrote:
> We've been running into a situation where exceptions in rdd.map() calls
> will
> not get recorded and shown on the web UI properly. We've discovered that
> this seems to occur because we're creating our own threads in
> foreachPartition() calls. If I have code like this:
>
>
>
> The tasks on the executors will fail because rdd1 will raise an exception
> for each record as we iterate across the "part" iterator inside the thread
> in the foreachPartition call. Usually, exceptions in Spark apps show up in
> the web UI on the application detail page, making problems easy to debug.
> However, if any exceptions get raised inside of these user threads, they
> don't show up in the web UI (instead, it just says that the executor was
> lost), and in the executor logs, we see errors like:
>
>
>
> What's going on here? Why are these exceptions not caught? And is there a
> way to have user threads register their exceptions properly?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Exceptions-in-threads-in-executor-code-don-t-get-caught-properly-tp24525.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
