marymwu created SPARK-16604:
-------------------------------
Summary: Spark2.0 fail in executing the sql statement which
includes partition field in the "select" statement while spark1.6 supports
Key: SPARK-16604
URL: https://issues.apache.org/jira/browse/SPARK-16604
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.0
Reporter: marymwu
Spark2.0 fail in executing the sql statement which includes partition field in
the "select" statement
error:
16/07/14 16:10:47 INFO HiveThriftServer2: set
sessionId(69e92ba1-4be2-4be9-bc81-7a00c5802ef8) to
exeId(c93f69b0-0f6e-4f07-afdc-ca6c41045fa3)
16/07/14 16:10:47 INFO SparkSqlParser: Parsing command: INSERT OVERWRITE TABLE
d_avatar.RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD_IMPALA
PARTITION(p_event_date='2016-07-13')
select
app_key,
app_version,
app_channel,
device_model,
total_num,
new_num,
active_num,
extant_num,
visits_num,
start_num,
p_event_date
from RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD where
p_event_date = '2016-07-13'
16/07/14 16:10:47 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:47 INFO ThriftHttpServlet: Cookie added for clientUserName hive
16/07/14 16:10:47 INFO HiveMetaStore: 108: get_table : db=default
tbl=rps__h_report_more_dimension_more_norm_first_channel_vcd
16/07/14 16:10:47 INFO audit: ugi=u_reaper ip=unknown-ip-addr
cmd=get_table : db=default
tbl=rps__h_report_more_dimension_more_norm_first_channel_vcd
16/07/14 16:10:47 INFO HiveMetaStore: 108: Opening raw store with implemenation
class:org.apache.hadoop.hive.metastore.ObjectStore
16/07/14 16:10:47 INFO ObjectStore: ObjectStore, initialize called
16/07/14 16:10:47 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:47 INFO ThriftHttpServlet: Cookie added for clientUserName hive
16/07/14 16:10:47 INFO Query: Reading in results for query
"org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is
closing
16/07/14 16:10:47 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is
MYSQL
16/07/14 16:10:47 INFO ObjectStore: Initialized ObjectStore
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO HiveMetaStore: 108: get_table : db=d_avatar
tbl=rps__h_report_more_dimension_more_norm_first_channel_vcd_impala
16/07/14 16:10:47 INFO audit: ugi=u_reaper ip=unknown-ip-addr
cmd=get_table : db=d_avatar
tbl=rps__h_report_more_dimension_more_norm_first_channel_vcd_impala
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: string
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:47 INFO CatalystSqlParser: Parsing command: bigint
16/07/14 16:10:49 WARN HiveSessionState$$anon$1: Max iterations (100) reached
for batch Resolution
16/07/14 16:10:49 ERROR SparkExecuteStatementOperation: Error executing query,
currentState RUNNING,
org.apache.spark.sql.AnalysisException: unresolved operator 'InsertIntoTable
MetastoreRelation d_avatar,
rps__h_report_more_dimension_more_norm_first_channel_vcd_impala, None,
Map(p_event_date -> Some(2016-07-13)), true, false;
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:39)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:56)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:309)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:51)
at
org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:125)
at
org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:51)
at
org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:56)
at
org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:48)
at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:62)
at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:541)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:673)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:213)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:157)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:154)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/07/14 16:10:49 ERROR SparkExecuteStatementOperation: Error running hive
query:
org.apache.hive.service.cli.HiveSQLException:
org.apache.spark.sql.AnalysisException: unresolved operator 'InsertIntoTable
MetastoreRelation d_avatar,
rps__h_report_more_dimension_more_norm_first_channel_vcd_impala, None,
Map(p_event_date -> Some(2016-07-13)), true, false;
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:248)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:157)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1$$anon$2.run(SparkExecuteStatementOperation.scala:154)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at
org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$1.run(SparkExecuteStatementOperation.scala:167)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/07/14 16:10:49 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:49 INFO ThriftHttpServlet: Cookie added for clientUserName hive
16/07/14 16:10:49 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:49 INFO ThriftHttpServlet: Cookie added for clientUserName hive
16/07/14 16:10:49 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:49 INFO ThriftHttpServlet: Cookie added for clientUserName hive
16/07/14 16:10:49 INFO ThriftHttpServlet: Could not validate cookie sent, will
try to generate a new cookie
16/07/14 16:10:49 INFO ThriftHttpServlet: Cookie added for clientUserName hive
Testcase:
spark sql: [
INSERT OVERWRITE TABLE
d_avatar.RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD_IMPALA
PARTITION(p_event_date='2016-07-13')
select
app_key,
app_version,
app_channel,
device_model,
total_num,
new_num,
active_num,
extant_num,
visits_num,
start_num,
p_event_date
from RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD where
p_event_date = '2016-07-13';
],
jdbcUrl:[jdbc:hive2://10.0.160.51:10000/default;transportMode=http;httpPath=cliservice;principal=spark/slave426.avatar.lenovomm....@avatar.lenovomm.com;hive.server2.proxy.user=u_avatar],
param:[StatParam [appName=avatar, appUser=u_avatar, execDate=2016-07-13,
processId=1893, taskId=43966]],
errorInfo:[org.apache.spark.sql.AnalysisException: unresolved operator
'InsertIntoTable MetastoreRelation d_avatar,
rps__h_report_more_dimension_more_norm_first_channel_vcd_impala, None,
Map(p_event_date -> Some(2016-07-13)), true, false;][home]
-------------------
d_avatar.RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD_IMPALA
RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD
CREATE TABLE
`RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD`(`app_key` string,
`app_channel` string, `app_version` string, `device_model` string, `total_num`
bigint, `new_num` bigint, `active_num` bigint, `extant_num` bigint,
`visits_num` bigint, `start_num` bigint)
PARTITIONED BY (`p_event_date` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.format' = '1'
)
STORED AS
INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
TBLPROPERTIES ('transient_lastDdlTime' = '1385178543');
CREATE TABLE
`d_avatar.RPS__H_REPORT_MORE_DIMENSION_MORE_NORM_FIRST_CHANNEL_VCD_IMPALA`(`app_key`
string, `app_version` string, `app_channel` string, `device_model` string,
`total_uv` bigint, `new_user` bigint, `uv` bigint, `extant_user` bigint, `pv`
bigint, `start_times` bigint)
PARTITIONED BY (`p_event_date` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('serialization.format' = '1'
)
STORED AS
INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
TBLPROPERTIES ('last_modified_by' = 'u_avatar',
'last_modified_time' = '1449840124',
'transient_lastDdlTime' = '1449840124');
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
