Sorry, I guess my code and the exception didn't make it to the mailing
list. Here's my code:
def main(args: Array[String]) {
val conf = new SparkConf().setAppName("Test app")
val sc = new SparkContext(conf)
val rdd = sc.parallelize(Array(1, 2, 3))
val rdd1 = rdd.map({x =>
[image: Inline image 1]
I'm not able to find the piece of code that you wrote, but you can use a
try...catch to catch your user specific exceptions and log it in the logs.
Something like this:
myRdd.map(x => try{ //something }catch{ case e:Exception =>
log.error("Whoops!! :" + e) })
Thanks
We've been running into a situation where exceptions in rdd.map() calls will
not get recorded and shown on the web UI properly. We've discovered that
this seems to occur because we're creating our own threads in
foreachPartition() calls. If I have code like this:
The tasks on the executors
