[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148798108 Are we certain that we only get `NoClassDefFoundError` but not also `ClassNotFoundException`? I wonder if we should just check for either of those in the match. Also cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user zjffdu commented on a diff in the pull request:
https://github.com/apache/spark/pull/9134#discussion_r42234736
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -655,6 +655,15 @@ object SparkSubmit {
// scalastyle:on println
}
System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
+ case e: NoClassDefFoundError =>
+e.printStackTrace(printStream)
+if (e.getMessage.contains("org/apache/hadoop/hive")) {
+ // scalastyle:off println
+ printStream.println(s"Failed to load hive class.")
+ printStream.println("You need to build Spark with -Phive and
-Phive-thriftserver.")
+ // scalastyle:on println
+}
+System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
--- End diff --
This issue happens only when SPARK_PREPEND_CLASSES is true. In this case
the spark assembly jar doesn't contain SparkSQLCLIDriver because spark is built
without -Phive-thriftserver, but it can still be found in CLASSPATH because
SPARK_PREPEND_CLASSES is true. But hive related class can not be found since it
is not in the spark assembly jar.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9134#discussion_r42231189
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -655,6 +655,15 @@ object SparkSubmit {
// scalastyle:on println
}
System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
+ case e: NoClassDefFoundError =>
+e.printStackTrace(printStream)
+if (e.getMessage.contains("org/apache/hadoop/hive")) {
+ // scalastyle:off println
+ printStream.println(s"Failed to load hive class.")
+ printStream.println("You need to build Spark with -Phive and
-Phive-thriftserver.")
+ // scalastyle:on println
+}
+System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
--- End diff --
I suspect it is roughly the same to the caller. I would like to understand
however how this happens?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/9134#discussion_r42121597
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -655,6 +655,15 @@ object SparkSubmit {
// scalastyle:on println
}
System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
+ case e: NoClassDefFoundError =>
+e.printStackTrace(printStream)
+if (e.getMessage.contains("org/apache/hadoop/hive")) {
+ // scalastyle:off println
+ printStream.println(s"Failed to load hive class.")
+ printStream.println("You need to build Spark with -Phive and
-Phive-thriftserver.")
+ // scalastyle:on println
+}
+System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
--- End diff --
Do we need to define another exit status to distinguish with the class not
found exception?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/9134#discussion_r42111793
--- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala ---
@@ -655,6 +655,15 @@ object SparkSubmit {
// scalastyle:on println
}
System.exit(CLASS_NOT_FOUND_EXIT_STATUS)
+ case e: NoClassDefFoundError =>
--- End diff --
I think this OK. (PS the exception isn't "unreadable" -- just
"uninformative"). `NoClassDefFoundError` is lower-level problem; classes were
compiled in the presence of some other classes that aren't available at
runtime. It's "worse" than `ClassNotFoundException`. It probably makes sense,
but is this really a problem of building *without* Hive support? I would kind
of think it's the opposite: you built with Hive and then it's not available at
runtime?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user zjffdu commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148356438 Check the test report. Seems the failed unit tests are unrelated. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148353599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43783/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148353596 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148353456 [Test build #43783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43783/console) for PR 9134 at commit [`5b6e651`](https://github.com/apache/spark/commit/5b6e6510dd3825910659cac95784f56c3ae9df51). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148326244 [Test build #43783 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43783/consoleFull) for PR 9134 at commit [`5b6e651`](https://github.com/apache/spark/commit/5b6e6510dd3825910659cac95784f56c3ae9df51). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148323656 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148323628 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148302621 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148302625 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43774/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148302613 [Test build #43774 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43774/console) for PR 9134 at commit [`2bdf979`](https://github.com/apache/spark/commit/2bdf97974be7bd6abdf55792b95a568eb423817d). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148301968 [Test build #43774 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/43774/consoleFull) for PR 9134 at commit [`2bdf979`](https://github.com/apache/spark/commit/2bdf97974be7bd6abdf55792b95a568eb423817d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148300064 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/9134#issuecomment-148300028 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-11125] [SQL] Unreadable exception when ...
GitHub user zjffdu opened a pull request: https://github.com/apache/spark/pull/9134 [SPARK-11125] [SQL] Unreadable exception when running spark-sql witho⦠â¦ut building with -Phive-thriftserver and SPARK_PREPEND_CLASSES is set This is the exception after this patch. ``` java.lang.NoClassDefFoundError: org/apache/hadoop/hive/cli/CliDriver at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:412) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:270) at org.apache.spark.util.Utils$.classForName(Utils.scala:173) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:647) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.cli.CliDriver at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 21 more Failed to load hive class. You need to build Spark with -Phive and -Phive-thriftserver. ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/spark SPARK-11125 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/9134.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #9134 commit 2bdf97974be7bd6abdf55792b95a568eb423817d Author: Jeff Zhang Date: 2015-10-15T07:02:05Z [SPARK-11125] [SQL] Unreadable exception when running spark-sql without building with -Phive-thriftserver and SPARK_PREPEND_CLASSES is set --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
