[jira] [Commented] (SPARK-16680) Set spark.driver.userClassPathFirst=true, and run spark-sql failed
[ https://issues.apache.org/jira/browse/SPARK-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15389168#comment-15389168 ] Dongjoon Hyun commented on SPARK-16680: --- Hi, [~KaiXinXIaoLei]. According to the log, it looks like you tried in local mode. That option is intended to be used for `cluster mode` only. http://spark.apache.org/docs/latest/configuration.html > Set spark.driver.userClassPathFirst=true, and run spark-sql failed > -- > > Key: SPARK-16680 > URL: https://issues.apache.org/jira/browse/SPARK-16680 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: KaiXinXIaoLei > > There is an exception when I run “sh bin/spark-sql --conf > spark.driver.userClassPathFirst=true” > 16/07/22 15:54:54 INFO HiveSharedState: Warehouse path is > 'file:/home/hll/code/spark-master-0722/spark-master/spark-warehouse'. > Exception in thread "main" java.lang.IllegalArgumentException: Unable to > locate hive jars to connect to metastore. Please set > spark.sql.hive.metastore.jars. > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:299) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive$lzycompute(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:288) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16680) Set spark.driver.userClassPathFirst=true, and run spark-sql failed
[ https://issues.apache.org/jira/browse/SPARK-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391701#comment-15391701 ] KaiXinXIaoLei commented on SPARK-16680: --- [~dongjoon] hi. I have tesed , there is still problem in yarn-cluster mode. My application is : val sparkConf = new SparkConf().setAppName("FemaleInfo") val sc = new SparkContext(sparkConf) val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) sqlContext.sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") sqlContext.sql("FROM src SELECT key, value").collect().foreach(println) There is a error log: 16/07/25 18:48:05 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: org/apache/hadoop/hive/shims/ShimLoader when creating Hive clien t using classpath: file:/opt/apache/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1469442803590_0003/container_1469442803590_0003_01_01/__app__.jar, file:/opt/apache/hadoop/tmp/nm-local-di r/usercache/root/appcache/application_1469442803590_0003/container_1469442803590_0003_01_01/datanucleus-api-jdo-3.2.6.jar, file:/opt/apache/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_14 69442803590_0003/container_1469442803590_0003_01_01/datanucleus-core-3.2.10.jar, file:/opt/apache/hadoop/tmp/nm-local-dir/usercache/root/appcache/application_1469442803590_0003/container_1469442803590_0 003_01_01/datanucleus-rdbms-3.2.9.jar Please make sure that jars for your version of hive and hadoop are included in the paths passed to SQLConfEntry(key = spark.sql.hive.metastore.jars, defaultValue=builtin, doc= Location of the jars that should be used to instantiate the HiveMetastoreClient. This property can be one of three options: " 1. "builtin" Use Hive 1.2.1, which is bundled with the Spark assembly jar when -Phive is enabled. When this option is chosen, spark.sql.hive.metastore.version must be either 1.2.1 or not defined. 2. "maven" Use Hive jars of specified version downloaded from Maven repositories. 3. A classpath in the standard format for both Hive and Hadoop. , isPublic = true). > Set spark.driver.userClassPathFirst=true, and run spark-sql failed > -- > > Key: SPARK-16680 > URL: https://issues.apache.org/jira/browse/SPARK-16680 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: KaiXinXIaoLei > > There is an exception when I run “sh bin/spark-sql --conf > spark.driver.userClassPathFirst=true” > 16/07/22 15:54:54 INFO HiveSharedState: Warehouse path is > 'file:/home/hll/code/spark-master-0722/spark-master/spark-warehouse'. > Exception in thread "main" java.lang.IllegalArgumentException: Unable to > locate hive jars to connect to metastore. Please set > spark.sql.hive.metastore.jars. > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:299) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive$lzycompute(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:288) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe,
[jira] [Commented] (SPARK-16680) Set spark.driver.userClassPathFirst=true, and run spark-sql failed
[ https://issues.apache.org/jira/browse/SPARK-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391702#comment-15391702 ] Dongjoon Hyun commented on SPARK-16680: --- Oh, I see. Thank you for providing details. > Set spark.driver.userClassPathFirst=true, and run spark-sql failed > -- > > Key: SPARK-16680 > URL: https://issues.apache.org/jira/browse/SPARK-16680 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: KaiXinXIaoLei > > There is an exception when I run “sh bin/spark-sql --conf > spark.driver.userClassPathFirst=true” > 16/07/22 15:54:54 INFO HiveSharedState: Warehouse path is > 'file:/home/hll/code/spark-master-0722/spark-master/spark-warehouse'. > Exception in thread "main" java.lang.IllegalArgumentException: Unable to > locate hive jars to connect to metastore. Please set > spark.sql.hive.metastore.jars. > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:299) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive$lzycompute(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:288) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16680) Set spark.driver.userClassPathFirst=true, and run spark-sql failed
[ https://issues.apache.org/jira/browse/SPARK-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15391709#comment-15391709 ] KaiXinXIaoLei commented on SPARK-16680: --- [~dongjoon] In the ChildFirstURLClassLoader private[spark] class ChildFirstURLClassLoader(urls: Array[URL], parent: ClassLoader) extends MutableURLClassLoader(urls, null) do you know why is not "extends MutableURLClassLoader(urls, parent)"? > Set spark.driver.userClassPathFirst=true, and run spark-sql failed > -- > > Key: SPARK-16680 > URL: https://issues.apache.org/jira/browse/SPARK-16680 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.1.0 >Reporter: KaiXinXIaoLei > > There is an exception when I run “sh bin/spark-sql --conf > spark.driver.userClassPathFirst=true” > 16/07/22 15:54:54 INFO HiveSharedState: Warehouse path is > 'file:/home/hll/code/spark-master-0722/spark-master/spark-warehouse'. > Exception in thread "main" java.lang.IllegalArgumentException: Unable to > locate hive jars to connect to metastore. Please set > spark.sql.hive.metastore.jars. > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:299) > at > org.apache.spark.sql.hive.HiveUtils$.newClientForMetadata(HiveUtils.scala:263) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive$lzycompute(HiveSharedState.scala:39) > at > org.apache.spark.sql.hive.HiveSharedState.metadataHive(HiveSharedState.scala:38) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive$lzycompute(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.HiveSessionState.metadataHive(HiveSessionState.scala:43) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:62) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.(SparkSQLCLIDriver.scala:288) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:137) > at > org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729) > at > org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org