[ https://issues.apache.org/jira/browse/SPARK-6187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-6187. ------------------------------ Resolution: Duplicate Ah, is this what was already resolved in https://issues.apache.org/jira/browse/SPARK-8625? > Report full executor exceptions to the driver > --------------------------------------------- > > Key: SPARK-6187 > URL: https://issues.apache.org/jira/browse/SPARK-6187 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.2.1 > Reporter: Piotr Kołaczkowski > Priority: Minor > > If the task fails for some reason, the driver seems to report only the > top-level exception, without the cause(s). While it is possible to recover > the full stacktrace from executor's logs, it is quite annoying and would be > better to just report the full stacktrace, with all the causes to the driver > application. > Example stacktrace I just got reported by the application: > {noformat} > org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 > in stage 0.0 failed 1 times, most recent failure: Lost task 5.0 in stage 0.0 > (TID 5, localhost): java.lang.NoClassDefFoundError: Could not initialize > class org.apache.cassandra.db.Keyspace > at > com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194) > at > com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:223) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:280) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:280) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > {noformat} > As you see, this is not very informative. > In fact, the real exception is: > {noformat} > java.lang.NoClassDefFoundError: Could not initialize class > org.apache.cassandra.db.Keyspace > at > com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194) > at > com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:227) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > java.lang.ExceptionInInitializerError > at > com.datastax.bdp.spark.writer.BulkTableWriter.writeSSTables(BulkTableWriter.scala:194) > at > com.datastax.bdp.spark.writer.BulkTableWriter.write(BulkTableWriter.scala:227) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284) > at > com.datastax.bdp.spark.writer.BulkTableWriter$BulkSaveRDDFunctions$$anonfun$bulkSaveToCassandra$1.apply(BulkTableWriter.scala:284) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61) > at org.apache.spark.scheduler.Task.run(Task.scala:56) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at > org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:741) > at org.apache.cassandra.db.Keyspace.<clinit>(Keyspace.java:72) > ... 10 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org