[
https://issues.apache.org/jira/browse/SPARK-21725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16233578#comment-16233578
]
xinzhang edited comment on SPARK-21725 at 11/1/17 2:31 AM:
-----------------------------------------------------------
[~mgaido]
1. hive 1.2.1
download a new tar only change hive-site.xml
about hive metastore with mysql . metastore(local 9083)
2.spark-sql copy the hive-site.xml
3.start spark-thriftserver
4.beeline connect the thriftserver
The metastore has changed from derby to mysql . My suggest is could u do it
with a new env. Without your current exit env.
Like what u say might be related to the metastore. I tested the case in
cdh5.7(hadoop2.6) and hadoop2.8(new env) , they will always appear , No
matter what I did . Hope your help . Thanks .
was (Author: zhangxin0112zx):
[~mgaido]
1. hive 1.2.1
download a new tar only change hive-site.xml
about hive metastore with mysql . metastore(local 9083)
2.spark-sql copy the hive-site.xml
3.start spark-thriftserver
4.beeline connect the thriftserver
The metastore has changed from derby to mysql . My suggest is could u do it as
a new env without your exit env.
Like what u say might be related to the metastore. I tested the case in
cdh5.7(hadoop2.6) and hadoop2.8(new env) , they will always appear , No
matter what I did . Hope your help . Thanks .
> spark thriftserver insert overwrite table partition select
> -----------------------------------------------------------
>
> Key: SPARK-21725
> URL: https://issues.apache.org/jira/browse/SPARK-21725
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.0
> Environment: centos 6.7 spark 2.1 jdk8
> Reporter: xinzhang
> Priority: Major
> Labels: spark-sql
>
> use thriftserver create table with partitions.
> session 1:
> SET hive.default.fileformat=Parquet;create table tmp_10(count bigint)
> partitioned by (pt string) stored as parquet;
> --ok
> !exit
> session 2:
> SET hive.default.fileformat=Parquet;create table tmp_11(count bigint)
> partitioned by (pt string) stored as parquet;
> --ok
> !exit
> session 3:
> --connect the thriftserver
> SET hive.default.fileformat=Parquet;insert overwrite table tmp_10
> partition(pt='1') select count(1) count from tmp_11;
> --ok
> !exit
> session 4(do it again):
> --connect the thriftserver
> SET hive.default.fileformat=Parquet;insert overwrite table tmp_10
> partition(pt='1') select count(1) count from tmp_11;
> --error
> !exit
> -------------------------------------------------------------------------------------
> 17/08/14 18:13:42 ERROR SparkExecuteStatementOperation: Error executing
> query, currentState RUNNING,
> java.lang.reflect.InvocationTargetException
> ......
> ......
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to move
> source
> hdfs://dc-hadoop54:50001/group/user/user1/meta/hive-temp-table/user1.db/tmp_11/.hive-staging_hive_2017-08-14_18-13-39_035_6303339779053
> 512282-2/-ext-10000/part-00000 to destination
> hdfs://dc-hadoop54:50001/group/user/user1/meta/hive-temp-table/user1.db/tmp_11/pt=1/part-00000
> at org.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2644)
> at org.apache.hadoop.hive.ql.metadata.Hive.copyFiles(Hive.java:2711)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1403)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:1324)
> ... 45 more
> Caused by: java.io.IOException: Filesystem closed
> ....
> -------------------------------------------------------------------------------------
> the doc about the parquet table desc here
> http://spark.apache.org/docs/latest/sql-programming-guide.html#parquet-files
> Hive metastore Parquet table conversion
> When reading from and writing to Hive metastore Parquet tables, Spark SQL
> will try to use its own Parquet support instead of Hive SerDe for better
> performance. This behavior is controlled by the
> spark.sql.hive.convertMetastoreParquet configuration, and is turned on by
> default.
> I am confused the problem appear in the table(partitions) but it is ok with
> table(with out partitions) . It means spark do not use its own parquet ?
> Maybe someone give any suggest how could I avoid the issue?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
