[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user navis closed the pull request at: https://github.com/apache/spark/pull/6858 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-155560038 @navis would you mind closing this patch if you don't intend to work on it? We can always reopen it later if there is more interest. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-149398204 @navis, would you mind addressing the comments in this PR or closing it if you don't plan to work on it soon? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-136894096 @navis are you actively working on this? Do you have time to address @vanzin's comments? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/6858#discussion_r38486014 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -121,6 +121,9 @@ case class SparkListenerApplicationStart( @DeveloperApi case class SparkListenerApplicationEnd(time: Long) extends SparkListenerEvent +@DeveloperApi +case class SparkListenerContextStop() extends SparkListenerEvent --- End diff -- +1 I would prefer to limit the number of new events --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/6858#discussion_r35340143
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/ClientWrapper.scala ---
@@ -401,4 +400,14 @@ private[hive] class ClientWrapper(
client.dropDatabase(db, true, false, true)
}
}
+
+ override def shutdown(): Unit = withHiveState {
+logInfo("Shutting down hive metastore")
+if (state._1 == Thread.currentThread()) {
--- End diff --
Will this ever be true?
This method is called from the `SparkListener` you're adding, which in a
live application is called from the listener bus dispatching thread, which is
most probably not the thread that created the `state` object.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/6858#discussion_r35339959 --- Diff: core/src/main/scala/org/apache/spark/scheduler/SparkListener.scala --- @@ -121,6 +121,9 @@ case class SparkListenerApplicationStart( @DeveloperApi case class SparkListenerApplicationEnd(time: Long) extends SparkListenerEvent +@DeveloperApi +case class SparkListenerContextStop() extends SparkListenerEvent --- End diff -- I'm not sure why you need this. Why won't `SparkListenerApplicationEnd` work? It's send in `SparkContext.stop()`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user markhamstra commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-122348869 I'd rather something more structured like case classes were used instead of plain tuples. It is less than immediately apparent what things like `state._2` mean. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112977799 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-11295 [Test build #35061 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35061/console) for PR 6858 at commit [`d65744d`](https://github.com/apache/spark/commit/d65744d265dc8212645703597dae5dce502b364f). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class SparkListenerContextStop() extends SparkListenerEvent` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112948147 [Test build #35061 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/35061/consoleFull) for PR 6858 at commit [`d65744d`](https://github.com/apache/spark/commit/d65744d265dc8212645703597dae5dce502b364f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112947982 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112947951 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112947680 ok to test @liancheng @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6858#issuecomment-112747266 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
GitHub user navis opened a pull request: https://github.com/apache/spark/pull/6858 [SPARK-8333] [SQL] Spark failed to delete temp directory created by HiveContext Shutdown underlying megastore and delete temporary directory when the hive context is closed. You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/spark SPARK-8333 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6858.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6858 commit d65744d265dc8212645703597dae5dce502b364f Author: navis.ryu Date: 2015-06-16T11:37:19Z [SPARK-8333] [SQL] Spark failed to delete temp directory created by HiveContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user navis commented on the pull request: https://github.com/apache/spark/pull/6840#issuecomment-112617257 Ignore this, for now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user navis closed the pull request at: https://github.com/apache/spark/pull/6840 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6840#issuecomment-112404110 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-8333] [SQL] Spark failed to delete temp...
GitHub user navis opened a pull request: https://github.com/apache/spark/pull/6840 [SPARK-8333] [SQL] Spark failed to delete temp directory created by HiveContext Added onContextStop(SparkListenerContextStop) method to SparkListener and registered the location of temporary directory to it which should be deleted when the context stops. You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/spark SPARK-8333 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/6840.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6840 commit 5d16ec7158d9aa353d80a6daaa60cff7d1ce011d Author: navis.ryu Date: 2015-06-16T11:37:19Z [SPARK-8333] [SQL] Spark failed to delete temp directory created by HiveContext --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
