We've been running into a situation where exceptions in rdd.map() calls will not get recorded and shown on the web UI properly. We've discovered that this seems to occur because we're creating our own threads in foreachPartition() calls. If I have code like this:
The tasks on the executors will fail because rdd1 will raise an exception for each record as we iterate across the "part" iterator inside the thread in the foreachPartition call. Usually, exceptions in Spark apps show up in the web UI on the application detail page, making problems easy to debug. However, if any exceptions get raised inside of these user threads, they don't show up in the web UI (instead, it just says that the executor was lost), and in the executor logs, we see errors like: What's going on here? Why are these exceptions not caught? And is there a way to have user threads register their exceptions properly? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Exceptions-in-threads-in-executor-code-don-t-get-caught-properly-tp24525.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
