[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-122035252 lgtm ! thanks for exploring all these options, I like this solution, huge improvement over the current behavior --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/7028 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-122142207 Thanks! Merged into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-121071447 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120799965 [Test build #37107 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37107/console) for PR 7028 at commit [`4714664`](https://github.com/apache/spark/commit/47146644e704a3d15661340a598c29b622cee28f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class GetArrayItem(child: Expression, ordinal: Expression) extends BinaryExpression ` * `case class GetMapValue(child: Expression, key: Expression) extends BinaryExpression ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-12083 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120776991 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120776986 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120777385 @squito, I have updated the PR and simplified it greatly, thus removing most of the magic. Now we simply append the current stack trace inside DAGScheduler, with no special marker. Only someone looking very closely at the trace would observe something amiss, and they would see the comment pointing to the associated JIRA. I think we should go with this approach over the wrapping approach for the simple reason that it is less likely to break callers. Some callers may fail to print the cause of the exception, which would have been fine before with no cause but now would not work. Some callers may inspect the message directly, which would be different (or we would have to duplicate). Some callers may display the exception to users, which would look significantly different if wrapped (rather than just made longer). Please let me know if you have further concerns; I have run into this problem sufficiently often now to want to push for this to go in (and the workaround is really annoying -- adding prints or try-catches to every line which may have caused the issue!). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120778600 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120778945 fetch failure - jenkins test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120779103 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120779099 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-120779285 [Test build #37107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/37107/consoleFull) for PR 7028 at commit [`4714664`](https://github.com/apache/spark/commit/47146644e704a3d15661340a598c29b622cee28f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-119291909 Hi @aarondav , sorry I had missed the part about breaking the api. It looks to me like the only place `JobWaiter.jobFailed` is called with a non-SparkException is when there is an [error submitting the job](https://github.com/apache/spark/blob/70beb808e13f6371968ac87f7cf625ed110375e6/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L794). This seems pretty minor -- would it technically be breaking the api contract to change to also be a SparkException? Or do you mean we break the api by just adding another wrapping SparkException, so the cause msg are different? That is true ... but the cause msg were pretty useless for programmatic use before, so as long as we're not violating some hard rule on compatibility, I also don't see that change as being a problem. all this said, I do still think what you've proposed is an improvement ... just want to make sure we've explored alternatives. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-118447192 The problem with any approach that wraps the exception is that we no longer throw an exception of the original type; we instead always throw SparkExceptions (as in your PR). This could be considered an API-breaking change, and one that would only break at runtime. The benefit of appending stack trace elements is that one can apply it to any exception without impact on the callers or DAGScheduler. I think that it is very intuitive to join the stacks before and after an event loop, and it has the expected semantics of code leaving the area I know and entering into scary Spark internals. However, I agree that the fact that we're joining a user-readable stack instead of the actual stack may be confusing. A compromise would be to make the stack look like this: ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh! at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1627) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1294) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1285) at = DAGScheduler EventLoop Submission =.() at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:558) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1741) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1759) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1774) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1788) at org.apache.spark.rdd.RDD.count(RDD.scala:1095) ... ``` The difference from the current version being that this also includes the Spark internals leading up to the actual EventLoop itself, which makes the stack appear more natural (but also uglier due to the several indirections through runJob). The other difference is that it would return a more intuitive (but less useful) stack trace in situations where callSite is currently used in Spark (such as when starting a new job in Spark Streaming). I believe that this change would satisfy your biggest concern (making the stack trace magical) because, as I said, I think this stack trace is actually very readable, and I'm willing to trade off some usefulness for less magic because it still gets us much farther than today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117692804 Hi @aarondav @andrewor14 , I would like to discuss this a little more. I think I didn't understand your proposal earlier and I didn't really express my concerns properly -- sorry I am going to backpedal on what I said before. My concern is not so much about the implementation details, but just how this looks to the end user. Certainly this change is better than the status quo. But I'd rather we add this the right way the first time, rather than put in something now, make users learn this new exception format, and then down the line we update it and users need to learn some new format. Understanding exceptions from Spark is already a big source of user confusion imo, so its worth spending a bit of time on this. There are 3 stack traces here: (a) from the executor where the original error occurred (b) where the exception was handled in `DAGScheduler.handleTaskSetFailed` as part of the event loop and (c) where the user code triggered the job with an action like `rdd.count()`. (b) is probably totally useless for most users, its just occasionally useful for a spark developer, but right now we give the user (a) + (b). This pr puts (b) + (c) together as if they are one stack trace. I definitely think that is an improvement -- users get to see (c), and mostly they'll just ignore (b) anyway so it mostly doesn't matter. But there will definitely be times that a curious user tries to understand the rest of the stack trace. Maybe they hit some spark bug and they want to try to understand it more before filing a jira, or they think that perhaps they are misusing spark and the stack trace will help them understand better, etc. In those cases, I think it'll cause a lot of confusion to have it appear that its all one stack trace. If we did have them separated as three different stack traces, it would be much easier for a user to see understand in that case, and I think they'd also be much more likely to look for an explanation of what the different parts are. (As I said earlier, I would not think I need to consult documentation to understand one simple stack trace.) I put together an alternative implementation here: https://github.com/apache/spark/pull/7156 (sadly the tests don't run b/c the scala compiler crashes running mima ... but its just meant for discussion, not to actually merge in any case, so just pretend the tests pass ...). That is an alternate way to get all the stack traces -- it makes (a) + (b) the cause of (c), so you get a stack trace like: ``` org.apache.spark.SparkException: job failed at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:558) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1741) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1759) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1774) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1788) at org.apache.spark.rdd.RDD.count(RDD.scala:1095) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38.apply$mcJ$sp(DAGSchedulerSuite.scala:883) ...[snip] at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh! at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$37$$anonfun$38$$anonfun$apply$mcJ$sp$2.apply(DAGSchedulerSuite.scala:883) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1627) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1774) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Driver stacktrace: at
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117375907 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117375849 [Test build #36197 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36197/console) for PR 7028 at commit [`a914cb0`](https://github.com/apache/spark/commit/a914cb03c7bf9b58e155ebc6f239d85545e2e593). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117342057 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117341985 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117342047 [Test build #36193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36193/consoleFull) for PR 7028 at commit [`a914cb0`](https://github.com/apache/spark/commit/a914cb03c7bf9b58e155ebc6f239d85545e2e593). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117342002 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117339579 Looks good, but now it doesn't merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117342051 [Test build #36193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36193/console) for PR 7028 at commit [`a914cb0`](https://github.com/apache/spark/commit/a914cb03c7bf9b58e155ebc6f239d85545e2e593). * This patch **fails some tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117343194 hey, sorry I haven't taken another look at this yet -- do you mind giving me a bit more time to look before merging? I should be able to look this evening, if not first thing tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117343740 Sure --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117353792 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117354476 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117354463 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-117354636 [Test build #36197 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/36197/consoleFull) for PR 7028 at commit [`a914cb0`](https://github.com/apache/spark/commit/a914cb03c7bf9b58e155ebc6f239d85545e2e593). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115561416 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115561383 [Test build #35824 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35824/console) for PR 7028 at commit [`19ab12e`](https://github.com/apache/spark/commit/19ab12ef36edb01f14caad0a713b7085536a5dc8). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115574020 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115570064 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115570079 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115737475 [Test build #35864 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35864/consoleFull) for PR 7028 at commit [`5fdb1cc`](https://github.com/apache/spark/commit/5fdb1cc4b077266577199542787f69f2d373c5f8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115737234 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115737217 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115821848 [Test build #35864 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35864/console) for PR 7028 at commit [`5fdb1cc`](https://github.com/apache/spark/commit/5fdb1cc4b077266577199542787f69f2d373c5f8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115821893 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115526307 I've updated the code to move the injection to runJob. This means that exceptions thrown from any path inside the DAGScheduler that fail the job will now have their stack trace supplemented, if the job was submitted via runJob (i.e., not an approximate job or submitted via SparkContext.submitJob). After looking over the usages of special call site, I found that streaming is a significant user and thus opted to keep the callsite parsing code. I could definitely see the benefit of removing the 100 lines of code or so that's required to do this for the sake of simplicity at the cost of streaming stack traces not being quite as helpful as they could be, so feel free to push back and I can change it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115528136 [Test build #35824 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35824/consoleFull) for PR 7028 at commit [`19ab12e`](https://github.com/apache/spark/commit/19ab12ef36edb01f14caad0a713b7085536a5dc8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115526912 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115526839 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user sujkh85 commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404750 NAVER - http://www.naver.com/ su...@naver.com ëê» ë³´ë´ì ë©ì¼ [spark] [SPARK-8644] Include call site in SparkException stack traces thrown by job failures (#7028) ì´ ë¤ìê³¼ ê°ì ì´ì ë¡ ì ì¡ ì¤í¨íìµëë¤. ë°ë ì¬ëì´ íìëì ë©ì¼ì ìì ì°¨ë¨ íììµëë¤. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
GitHub user aarondav opened a pull request: https://github.com/apache/spark/pull/7028 [SPARK-8644] Include call site in SparkException stack traces thrown by job failures Example exception (new part at bottom, clearly demarcated): ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh! at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34$$anonfun$apply$mcJ$sp$1.apply(DAGSchedulerSuite.scala:851) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34$$anonfun$apply$mcJ$sp$1.apply(DAGSchedulerSuite.scala:851) at scala.collection.Iterator$$anon$11.next(Iterator.scala:328) at org.apache.spark.util.Utils$.getIteratorSize(Utils.scala:1637) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.rdd.RDD$$anonfun$count$1.apply(RDD.scala:1095) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1765) at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1765) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:63) at org.apache.spark.scheduler.Task.run(Task.scala:70) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:213) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1285) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1276) at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1275) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1275) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:749) at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:749) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:749) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1486) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) at = Job Submission =.(Native Method) at org.apache.spark.rdd.RDD.count(RDD.scala:1095) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34.apply$mcJ$sp(DAGSchedulerSuite.scala:851) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34.apply(DAGSchedulerSuite.scala:851) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34.apply(DAGSchedulerSuite.scala:851) at org.scalatest.Assertions$class.intercept(Assertions.scala:997) at org.scalatest.FunSuite.intercept(FunSuite.scala:1555) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33.apply$mcV$sp(DAGSchedulerSuite.scala:850) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33.apply(DAGSchedulerSuite.scala:849) at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33.apply(DAGSchedulerSuite.scala:849) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:42) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) ``` Major complexity was getting the actual stack trace which we already had in the right place back into StackTraceElement form. You can merge this pull request into a Git repository by running: $ git pull
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115413847 This will be great to add that to the exception msg, the current Driver stacktrace is not very useful. But, I think just munging it with the existing stack trace might be really confusing to spark users. Java stack traces have very standard interpretations (part of the reason they are so useful). Eg., if I saw this line `at = Job Submission =.(Native Method)` I would just probably naively assume spark was calling some magical native method, but that it was still a normal call stack (and I wouldn't think I need to look in the docs to interpret a stack trace). How about instead just making it a separate section in the msg? eg. ``` org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 1 times, most recent failure: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.RuntimeException: uh-oh! at org.apache.spark.scheduler.DAGSchedulerSuite$$anonfun$33$$anonfun$34$$anonfun$apply$mcJ$sp$1.apply(DAGSchedulerSuite.scala:851) ... Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1285) ... Job Submission stacktrace: at org.apache.spark.rdd.RDD.count(RDD.scala:1095) ... ``` Then you could also just do this in `JobWaiter` or `sc.runJob`, which would make it much simpler. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115423583 [Test build #35813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35813/consoleFull) for PR 7028 at commit [`4725d14`](https://github.com/apache/spark/commit/4725d14c224fd42bd49d89ebb492c9f2e834b5af). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115405169 [Test build #35812 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35812/console) for PR 7028 at commit [`714be86`](https://github.com/apache/spark/commit/714be86b91683c08e81eb0a3bc0cdc6172e42076). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115423231 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115423222 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115405172 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115422594 I think the most important thing is to include the user stack trace somewhere. Users don't really care what's going on as long as they can identify a line of code from their codebase in the stack. That said, I see your point about the current approach being overly magical. Note that producing an Exception that matches your example would require wrapping/replacing whichever actual exception with a SparkException (otherwise we cannot rewrite the message). However, we do have two disjoint options: 1. Keep the munging logic in DAGScheduler.failJobAndIndependentStages and simply put the Job Submission stack trace above the driver stack trace, included in the Exception's message. This puts the exception a bit out of order but would avoid having to parse the longForm CallSite. We can do this because this method only throws SparkExceptions anyway. 2. Move the munging logic to DAGScheduler.runJob and keep it as a fake part of the stack trace. This would allow us to inject the user stack trace into *all* exceptions thrown as part of the DAGScheduler EventLoop rather than just ones that induce normal stage fails. With number 2, we could additionally replace the use of callSite parsing with the current thread's stack trace, which would make it look a lot less magical (it's just bridging the two real stack traces across the event loop). I'd personally prefer 2 for its improved scope, but I don't know enough about how the call site may differ from the physical stack trace to say if we should also just use the local stack. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404852 [Test build #35812 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35812/consoleFull) for PR 7028 at commit [`714be86`](https://github.com/apache/spark/commit/714be86b91683c08e81eb0a3bc0cdc6172e42076). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user sujkh85 commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404783 NAVER - http://www.naver.com/ su...@naver.com ëê» ë³´ë´ì ë©ì¼ Re: [spark] [SPARK-8644] Include call site in SparkException stack traces thrown by job failures (#7028) ì´ ë¤ìê³¼ ê°ì ì´ì ë¡ ì ì¡ ì¤í¨íìµëë¤. ë°ë ì¬ëì´ íìëì ë©ì¼ì ìì ì°¨ë¨ íììµëë¤. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404657 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404677 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115404640 cc @JoshRosen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user squito commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115466021 OK, I took a closer look at the the way the exception handling works and I see what you mean. I'm not particularly tied to that formatting I suggested, that was just an example, totally fine to do something else that is similar. I agree that as long as the user can clearly identify their code in the driver, and their code in the executors, that is a big help. I think either of your proposed solutions sound fine -- I'm also a little partial to (2) if its simpler. I am not sure I understand it 100%, but sounds good :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115446599 [Test build #35813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35813/console) for PR 7028 at commit [`4725d14`](https://github.com/apache/spark/commit/4725d14c224fd42bd49d89ebb492c9f2e834b5af). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8644] Include call site in SparkExcepti...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7028#issuecomment-115446658 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
