[jira] [Commented] (SPARK-26421) Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode

2018-12-22 Thread thinktothings (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727789#comment-16727789
 ] 

thinktothings commented on SPARK-26421:
---

Thanks, I understand this question, I have another question, I don't understand 
how Spark 2.4.0 DataSet.show submits the job, can you help me? Give me relevant 
information to learn?Spark1.6 rdd is aware of the process of submitting jobs, 
and has seen the implementation of the source code in detail.

> Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea 
> local mode
> -
>
> Key: SPARK-26421
> URL: https://issues.apache.org/jira/browse/SPARK-26421
> Project: Spark
>  Issue Type: Bug
>  Components: Deploy
>Affects Versions: 2.4.0
> Environment: ).idea maven project
> ).jdk 1.8.0_191
> ).hadoop 3.1.1
> ).spark 2.4.0
>Reporter: thinktothings
>Priority: Major
>
> ).Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea 
> local mode
> ).idea maven project
> ).spark.sql connect hive 
> val spark = SparkSession
>  .builder()
>  .master("local")
>  .appName("Spark Hive Example")
>  .config("spark.sql.warehouse.dir", warehouseLocation)
>  .enableHiveSupport()
>  .getOrCreate()
> spark.sql("show databases").show()
>  
> ).do this   error ,local environment not cluser
> --
> Exception in thread "main" java.lang.ExceptionInInitializerError
>  at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:105)
>  at java.lang.Class.forName0(Native Method)
>  at java.lang.Class.forName(Class.java:348)
>  at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
>  at 
> org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
>  at 
> org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
>  at 
> com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run$.main(Run.scala:19)
>  at 
> com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run.main(Run.scala)
> Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major 
> version number: 3.1.1
>  at 
> org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
>  at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
>  at 
> org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
>  at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368)
>  ... 8 more
> Process finished with exit code 1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26421) Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea local mode

2018-12-22 Thread Dongjoon Hyun (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16727737#comment-16727737
 ] 

Dongjoon Hyun commented on SPARK-26421:
---

Hi, [~thinktothings]. This is a duplication of SPARK-23534. 
Apache Spark officially doesn't support Hadoop 3.

> Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea 
> local mode
> -
>
> Key: SPARK-26421
> URL: https://issues.apache.org/jira/browse/SPARK-26421
> Project: Spark
>  Issue Type: Bug
>  Components: Deploy
>Affects Versions: 2.4.0
> Environment: ).idea maven project
> ).jdk 1.8.0_191
> ).hadoop 3.1.1
> ).spark 2.4.0
>Reporter: thinktothings
>Priority: Major
>
> ).Spark2.4.0 integration hadoop3.1.1 causes hive sql not to use，just in idea 
> local mode
> ).idea maven project
> ).spark.sql connect hive 
> val spark = SparkSession
>  .builder()
>  .master("local")
>  .appName("Spark Hive Example")
>  .config("spark.sql.warehouse.dir", warehouseLocation)
>  .enableHiveSupport()
>  .getOrCreate()
> spark.sql("show databases").show()
>  
> ).do this   error ,local environment not cluser
> --
> Exception in thread "main" java.lang.ExceptionInInitializerError
>  at org.apache.hadoop.hive.conf.HiveConf.(HiveConf.java:105)
>  at java.lang.Class.forName0(Native Method)
>  at java.lang.Class.forName(Class.java:348)
>  at org.apache.spark.util.Utils$.classForName(Utils.scala:238)
>  at 
> org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
>  at 
> org.apache.spark.sql.SparkSession$Builder.enableHiveSupport(SparkSession.scala:866)
>  at 
> com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run$.main(Run.scala:19)
>  at 
> com.opensource.bigdata.spark.sql.n_10_spark_hive.n_01_show_database.Run.main(Run.scala)
> Caused by: java.lang.IllegalArgumentException: Unrecognized Hadoop major 
> version number: 3.1.1
>  at 
> org.apache.hadoop.hive.shims.ShimLoader.getMajorVersion(ShimLoader.java:174)
>  at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:139)
>  at 
> org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
>  at org.apache.hadoop.hive.conf.HiveConf$ConfVars.(HiveConf.java:368)
>  ... 8 more
> Process finished with exit code 1



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org